On-board tool tracking system and methods of computer assisted surgery

ABSTRACT

A number of improvements are provided relating to computer aided surgery utilizing an on tool tracking system. The various improvements relate generally to both the methods used during computer aided surgery and the devices used during such procedures. Other improvements relate to the structure of the tools used during a procedure and how the tools can be controlled using the OTT device. Still other improvements relate to methods of providing feedback during a procedure to improve either the efficiency or quality, or both, for a procedure including the rate of and type of data processed depending upon a CAS mode.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation-in-part of International ApplicationNumber PCT/US2012/044486, filed Jun. 27, 2012, titled “ON-BOARD TOOLTRACKING SYSTEM AND METHODS OF COMPUTER ASSISTED SURGERY,” which claimspriority to U.S. Provisional Patent Application No. 61/501,489, filedJun. 27, 2011, titled “SYSTEM FOR COMPUTER ASSISTED NAVIGATION ANDCONTROL OF A POWER TOOL,” each of which is incorporated by reference inits entirety for all purposes.

STATEMENT AS TO FEDERALLY SPONSORED RESEARCH

This invention was made with Government support under Grant No. 0578104,awarded by the Department of Defense. The Government has certain rightsin the invention.

INCORPORATION BY REFERENCE

All publications and patent applications mentioned in this specificationare herein incorporated by reference to the same extent as if eachindividual publication or patent application was specifically andindividually indicated to be incorporated by reference.

FIELD

The present invention relates to the field of computer assisted surgery.Specifically, the present invention relates to various aspects of asurgical suite in which a tracking system on a tool provides guidance orassistance during a surgical procedure.

BACKGROUND

Many surgical procedures are complex procedures requiring numerousalignment jigs and intricate soft tissue procedures. Preparing andplacing the alignment jigs and other preparation is often a significantpart of the procedure and involves various errors. For instance, whenperforming a total knee replacement procedure (“TKR”), the prosthesismust be accurately implanted to ensure that the joint surfaces areproperly aligned. If the alignment is inaccurate, the misalignment cancompromise the function and eventually lead to failure of the joint,requiring the complex task of replacing one or more portions of the kneeprosthesis.

To ensure that the prosthesis is accurately implanted, during a TKRprocedure, the surgeon uses a variety of jigs to guide the cutting ofthe femur, the tibia and sometimes the patella. The jigs are complex andexpensive devices that require significant time and skill to locate andattach on the patient during the surgical procedure.

The advent of computer assisted surgery (CAS) provides the promise ofsimplifying many of the complexities of surgical procedures. To datesystems have been developed that utilize separate room based trackingsystems designed to monitor the cutting jigs, tools and the patient. Insome instances, the computer may be used to guide the surgeon during theprocess. The placement of the in room camera closer to the tool has beenproposed. However, improvements are needed to address the challenges ofthe line of sight requirements and other real-time and dynamicenvironment of a surgical procedure.

Although computer assisted surgery holds promise, there are numerousaspects to be addressed to make a system commercially viable and usefulto surgeons. There continues to exist numerous aspects of computerassisted surgery that require improvement to improve the efficiencyand/or quality of the procedure for processing of CAS data, and moreuseful outputs to the user.

SUMMARY OF THE DISCLOSURE

In one aspect, a tactile feedback mechanism includes a first platform; asecond platform; a scissor linkage formed by a first linkage coupled toa second linkage, the scissor linkage extending between the firstplatform and the second platform wherein a first end of the firstlinkage is coupled to the first platform and a second end of the firstlinkage is coupled to the second platform and the first end of thesecond linkage is coupled to the first platform and the second end ofthe second linkage is coupled to the second platform; and at least oneposition restoration element coupled to the scissor linkage to adjust aforce response of the relative movement between the first platform andthe second platform. In some aspects the at least one positionrestoration element is coupled between the first end of the firstlinkage and the second end of the second linkage. In another aspect, theat least one position restoration element extends along a secondplatform and is coupled to the scissor linkage to adjust the movement ofthe second linkage second end relative to the second platform. In oneembodiment, the first platform and the second platform are configuredfor operation alongside, partially covering, partially surrounding,partially over or completely over an on/off and or speed control triggerof a surgical tool. In one embodiment, a trigger cover is placed withinthe first platform for engagement with the trigger.

In still another configuration of a tactile feedback mechanism, there isprovided at least one position restoration element coupled to thescissor linkage to adjust a force response of the relative movementbetween the first platform and the second platform is coupled so as toextend between the first platform and the second platform. Stillfurther, there may be provided a position restoration element coupled tothe scissor linkage and extending along the second platform. In onespecific configuration of a tactile feedback mechanism, the positionrestoration element is a return spring coupled to the second end of thesecond linkage and there is an override spring coupled to the returnspring and also there may be an actuator coupled to the override spring.In another embodiment of a tactile feedback mechanism, the positionrestoration element is a spring coupled in tension to the movement ofthe second ends of the scissor linkage relative to the second platform.In still another position restoration element configuration, a springcoupled in compression to the movement of the second ends of the scissorlinkage relative to the second platform. In some feedback mechanisms,there is also a shaft extending from an opening in the second platformand coupled to the scissor linkage wherein movement of the scissorlinkage produces corresponding movement of the shaft relative to theopening. The alternatives to the shaft include for example, a flexibleshaft portion, a cable portion, a hollow shaft portion or a flexiblelinkage portion.

In still other configurations, an embodiment of a tactile feedbackmechanism may be used in conjunction with an embodiment of an on tooltracking (OTT) device configured for use in computer assisted surgery.Such an OTT device would include for example a component or series ofcomponents working in cooperation within the on tool tracking devicethat are adapted and configured to translate the shaft relative movementinto a signal used in a computer assisted surgery procedure. In oneaspect the component may be an actuator, a solenoid, a motor, apotentiometer, a linear potentiometer, and inductive position sensor, ora linear encoder or other device positioned adjacent to the cable toregister and measure displacement of the cable. In one aspect, cablemovement relates to a signal indicative of the operation of the triggerof the surgical tool. In still further embodiments, the same componentor a different component may also act as an actuator to impart movementto the shaft to influence the relative movement between the firstplatform and the second platform. These various components and functionsare each used in support of being configured to impart movement to orrespond to the shaft in response to a signal related to controlling theoperation of the surgical tool during a computer assisted surgeryprocedure.

In another embodiment, there is provided a reference frame for use in acomputer assisted surgery procedure with navigation. The reference frameincludes a frame having a flat or 3-dimensional surface or cluster ofmarkers bounded or unbounded by perimeter; the stem extending from theframe; a coupling on the stem; a base having a first surface configuredto engage a portion of the anatomy within a surgical field related tothe procedure and a second surface to engage with the coupling. In someconfigurations, there may also be provided at least one registrationelement on the coupling and at least one registration element on thesecond surface wherein the registration elements are adapted andconfigured for mating cooperation at one or more repeatable 3D relativepositions and orientations when the coupling is engaged to the secondsurface. In still further configurations, a plurality of registrationelements on the coupling; and a plurality of registration elements onthe second surface, wherein a portion of the registration elements onthe coupling when engaged with a portion of the registration elements onthe second surface will orient the frame in a first orientation withinthe surgical field. In one aspect, movement between the coupling in thesecond surface to engage other of said plurality of registrationelements will position the frame in a second, different orientationwithin the surgical field. In some aspects, the first and secondorientations are known position and are used in surgical preplanning.The reference frame may include other features such as surface forengagement anatomy, and aperture for a fixation element orconfigurations to mate with particular anatomical targets. In anotheraspect, there is provided a reference frame according to claim C1,further comprising: a reference frame guide having a frame and a stemextending from the frame, wherein the stem has a curvature or shapeconfigured to engage with an anatomical feature to assist in theplacement of the reference frame. In one aspect, the reference frameguide further comprising: one or more engagement elements along theframe for temporary engagement with the perimeter or a portion of thereference frame to permit proper positioning and adjustment of a baseassociated with the reference frame. In one aspect, the portion of thebony anatomy relates to the placement of the stem in relation to thecondyles. In another aspect, the reference frame includes a mountcoupling adapted and configured to maintain the relative position andorientation of the coupling and the second surface. In one aspect, themount coupling is provided in the reference frame such that when themount coupling is mated to the base the mount coupling is within aninterior portion of the reference frame. In another aspect, the mountcoupling is provided in the reference frame such that when the mountcoupling attached to the reference frame the mount couplingsubstantially or completely surrounds the area of mating contact betweenthe coupling and the second surface.

In one alternative embodiment, there is provided a method of performinga computer aided surgery procedure within a surgical field. First, stepof attaching a first reference frame within the surgical field at afirst position; then, attaching a second reference frame within thesurgical field at a second position; and thereafter initiating an activestep of the procedure using the surgical tool while maintainingpositioning information used during the computer aided surgery procedureobtained from both the first and the second reference frames. In onealternative aspect, there is the step of adjusting the position of asurgical tool relative to a section of the anatomy during a step or aspart of the procedure while maintaining positioning information usedduring the computer aided surgery procedure obtained from the firstand/or the second reference frames attached to the section of theanatomy. In one alternative embodiment there is also the step ofhovering the surgical tool during a step as part of the procedure whilemaintaining positioning information used during the computer aidedsurgery procedure obtained from either the first and/or the secondreference frames. In still further aspect, there are methods includingone or more of the steps of initiating, adjusting or hovering areperformed in furtherance of one or more steps of a computer assistedsurgery procedure on a knee. In a still further alternative, there aremethods including, one or more steps of a computer assisted surgeryprocedure on a knee comprising: making a single distal condyle cut, orseparate distal medial and lateral condyle cuts, making an anterior cut,making a posterior lateral condyle cut, making a posterior medialcondyle cut, making an anterior chamfer cut, making a posterior lateralcondyle chamfer cut, making a posterior medial condyle chamfer cutmaking a femoral box cut, drilling one or more holes in a portion of asurgical site and making a tibial proximal cut and associated holes orcuts for knee tibial component fixation anchoring features. In stillanother alternative embodiment, the method proceeds while maintainingthe first reference frame and the second reference frame in the firstposition and the second position respectively after completion of theattaching steps, altering the orientation of a portion of the referenceframe relative to the surgical field and thereafter using positioninformation from the altered orientation for a portion of a computeraided surgery procedure. In still further aspect, the positioninformation relating to the orientations of the first reference frameand the second reference frame in both the initial and the alteredorientation are used as part of the preplanning processes for thecomputer aided surgery.

In another alternative embodiment, there is an on tool tracking andguidance device. In one aspect, the device has a housing having asurface or feature for releasable engagement with a portion of asurgical tool; a first camera and, optionally, a second camera in astereo-vision arrangement where each of the first camera and the secondcamera (if provided) provides an image output selected for viewingsubstantially all or a portion of a surgical field selected for acomputer assisted surgery procedure. The OTT device in one aspect mayinclude a simple output device for communicating information to the userabout the ongoing OTT CAS processes. In still other aspects, the OTTdevice may include a separate or onboard projector configured to providean output at least partially within the surgical field of view. Thevarious embodiments of OTT device is described herein may incorporate awide variety of capabilities for electronic image processing and imagecommunication capabilities within the housing. Still further, additionalembodiments may be configured to receive an output from each of the one,two, or more cameras provided by an embodiment of an OTT device.Additionally or optionally, electronics and processing capabilities ofthe OTT device may be utilized to perform a wide range of digitalprocessing functions. In one aspect, electronics included with the OTTperform an image processing operation using at least a portion of theoutput from one or both cameras configured for use in the computerassisted surgery procedure. In one aspect, the camera selected for usewith an OTT device may include a field of view from about 70 mm to about200 mm, or optionally, from about 40 mm to 250 mm from the first andsecond cameras. Still other ranges and camera configurations may be usedin various other embodiments.

In a still further embodiment, the OTT housing surface for releasableengagement with a portion of a surgical tool is shaped to form acomplementary curve with the portion of the surgical tool or a modifiedsurgical tool selected for engagement with the housing and, in someinstances, part of the surgical tool is modified to accommodatereleasable engagement with the housing surface. In one example, thesurface for releasable engagement with a portion of a surgical tool isadapted and configured so that when the surface is coupled to thesurgical tool at least a portion of an active segment of the surgicaltool lies within the horizontal field of view and the vertical field ofview.

In still further aspects, the onboard or separate projector may includesuch attributes as: the output from the projector is projected on ornear an active element associated with a surgical tool attached to thehousing; the output from the projector is adapted for projection on aportion of the patients anatomy such as the bone and/or surroundingtissue, or on or within the surgical field surface in the surgicalscene; an adaptation process gives an adapted projector output that isadjusted for the curvature, roughness or condition of the anatomy. Inone aspect, the projector is what is known as a pico projector.

In on embodiment, there is a method for performing a computer assistedsurgery procedure using a hand held surgical instrument having an ontool tracking device attached thereto including collecting andprocessing computer assisted surgery data using the on tool trackingdevice; assessing the data in real time during the computer assistedsurgery procedure; performing CAS related operations using the on tooltracking device selected from at least two of: controlling the operationof the tool, controlling the speed of the tool and providing to the userguidance related to a CAS step; controlling the operation or speed ofthe tool or providing guidance to the user to adjust the speed of thetool; and providing a user of the surgical instrument an output relatedto the assessing step. There may also be, in additional or alternativeaspects, one or more of displaying, projecting, or indicating an outputrelated to a computer assisted surgery processing step.

There may also be, in additional or alternative aspects, an outputcomprising one or more of a tactile indication, a haptic indication, anaudio indication or a visual indication; the tactile indicationcomprises a temperature indication; and the haptic indication comprisesa force indication or a vibration indication. Still further aspects, theoutput is the control signal automatically generated to adjust aperformance parameter of the surgical tool in response to a result ofthe assessing step. In other aspects, the performance parameter includesmodifying a tool cutting speed or stopping a tool operation. The outputof providing a step further comprising electronics to control operationof power tools (modifying cutting speed and/or stopping it). There mayalso be, in additional or alternative aspects, a determining step thatis based upon an evaluation of one or more of: a physical parameterwithin the surgical field such as position or combination of positionsof elements tracked in the field through reference frames attached tothem a reference frame input, projected image(s) taken, a motiondetected from a sensor, a motion detection from a calculation, theoverall progress of a computer aided surgery procedure, and a measuredor predicted deviation from a previously prepared computer aided surgeryplan. Still further, the determining step selects one of a number ofpredefined processing modes, such as for example hover mode, siteapproach mode, and active step mode. In each of these modes there arespecific outputs, processing techniques and algorithms applied to theCAS data.

In still further aspects, there are OTT CAS processing mode factorsselected from one or more of: a camera frame size; an OTT cameraorientation; an adjustment to a camera software program or firmware inaccordance with the desired adjustment; adjustments to an OTT camera orother camera image outputs to modify a size of a region of interestwithin a horizontal field of view, the vertical field of view or boththe horizontal and the vertical fields of view of the camera; drivesignals for adjustable camera lens adjustment or positioning; imageframe rate; image output quality; refresh rate; frame grabber rate;reference frame two; reference frame one; on reference frame fiducialselect; off reference frame fiducial select; visual spectrum processing;IR spectrum processing; reflective spectrum processing; LED orillumination spectrum processing; surgical tool motor/actuator speed anddirection, overall CAS procedure progress; specific CAS step progress;image data array modification; an OTT pico projector refresh rate; anOTT pico projector accuracy; one or more image segmentation techniques;one or more logic-based extractions of an image portion based on a CASprogress; signal-to-noise ratio adjustment; one or more imageamplification process, one or more imaging filtering process; applyingweighted averages or other factors for dynamic, real-time enhancement orreduction of image rate, pixel or sub-pixel vision processing; a handtremor compensation; an instrument-based noise compensation for a saw, adrill or other electrical surgical tool and a vibration compensationprocess, and any user preferences through the LCD touch screen if onewas onboard the OTT device along with any one or more additional OTT CASprocessing mode factors, based on information from the OTT each alone orin any combination.

In still other aspects, the output is provided to the user with aprojector in the on tool tracking device. In addition, the projectoroutput is automatically or manually adjusted based upon a physicalcharacteristic of the surgical site presented during the display of theprojector output. It is to be appreciated that the physicalcharacteristic is one or more of the shape of the portion of the sizeavailable to the projector output; the topography in the projectorprojected field and the orientation of the projector to the portion ofthe site available for the projector output. Optionally, the projectoror a display on the OTT device has an output that includes informationvisible to the user of the surgical tool while the surgical tool is inuse in the surgical site. In still further aspects, the projector or adisplay on the OTT device output includes information visible to theuser of the surgical tool to indicate the position, relative motion,orientation, or other navigation or guidance parameter related to thepositioning of the active element of the surgical tool within thesurgical field according to the surgical plan. Still the step ofproviding an output from an OTT device may include displaying the outputon a system screen; on a GUI interface on the OTT or a mobile devicescreen.

In a still further aspect, any of the above steps of outputting a CAS orguidance output to the user is, optionally, changed and an OTT CASprocessing technique or output is modified as a result of the userperforming one or more steps of a computer assisted surgery procedure ona knee comprising: making one or more distal femur cuts, making a distalfemur anterior cut, making a distal femur posterior lateral condyle cut,making a distal femur posterior medial condyle cut, making a distalfemur anterior chamfer cut, making a distal femur posterior lateralcondyle chamfer cut, making a distal femur posterior medial condylechamfer cut, making proximal tibial cut or any tibial holes or cuts tocater for any anchors fixation features on a tibial component such aspegs, stems, keels, etc. In still other alternatives, the methods hereinof outputting a CAS output to the user is changed as a result of one ofthe above recited steps performed during a surgical procedure related toone of a shoulder; a hip; an ankle; a vertebra; an elbow or deformitycorrection or fracture reduction bone osteotomy. Additionally, an OTTCAS processing technique or output is modified as a result of one of theabove recited steps performed during a surgical procedure related to oneof a shoulder; a hip; an ankle; a vertebra; an elbow or deformitycorrection or fracture reduction bone osteotomy.

In still another aspects, there is provided a system for performingcomputer assisted surgery, having a surgical tool having an activeelement corresponding to the surgical function of the tool; an on tooltracking device coupled to the tool using a housing configured to engagewith at least a portion of the surgical tool; at least one camera in thehousing configured to obtain imaging information related to the surgicaltool and a surgical field; an output device like a graphical screendisplay, or, optionally a projector in the housing configured to providea projected output on or near an active element of the surgical tool; acomputer having computer readable instructions stored within electronicmemory for performing a computer assisted surgical procedure using dataat least partially obtained from the on tool tracking device and toprovide an output for use during a step of the surgery. When the systemincludes a projector within the OTT capabilities, the projector furthercomprising one or more of the following: projection capability toproject an output on a portion of the patient's anatomy, a surfacewithin the surgical scene, an electronic device, or other object withinthe projector output range. In one configuration, the computer is in thehousing. In another the computer is separated from the on tool trackingdevice and connected via a wired or a wireless connection. In stillfurther aspects, the system includes one or more of the computerreadable instructions for performing any of the CAS mode select methodsdescribed above. In still further aspect, the system may include the ontool tracking device having one or more of the elements described above.The system may adapted and configured for use with one or more referenceframes and associated methods described herein. In a still furtheraspect, the system is adapted and configured for use in combination witha tactile feedback mechanism described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe claims that follow. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1 illustrates an isometric view of an example of an on tooltracking device attached to a surgical instrument.

FIG. 2 illustrates an isometric view of an on tool tracking deviceattached to a surgical instrument.

FIG. 3 illustrates an isometric view of the on tool tracking device ofFIG. 1 with a cover removed to show internal components.

FIG. 4 illustrates an isometric view of the on tool tracking device ofFIG. 2 with a cover removed to show internal components.

FIG. 5 illustrates a top down view of the on tool tracking device ofFIG. 4

FIG. 6 illustrates an isometric view of the on tool tracking device ofFIG. 5 separated from the surgical tool

FIG. 7 illustrates electronics package and control circuitry visible inFIGS. 5 and 6 but in this view is removed from the OTT housing.

FIGS. 8A, 8B, 9, and 10 provide graphical information relating to thechanges in camera field based on camera angle in some OTT deviceconfigurations.

FIGS. 11A, 11B, 11C and 11D provide additional information relating tovariations of camera angle.

FIGS. 12A and 13A provide side and isometric views respectively of aprojector used with an on tool tracking device.

FIGS. 12B, 13B and 13C provide side, isometric and top viewsrespectively of a projector in an angled orientation in use with an ontool tracking device.

FIGS. 14A, 14B, 15A, and 15B each illustrate schematic views of severaldifferent electronic component configurations used by some on tooltracking device embodiments.

FIGS. 16A, 16B and 16C illustrate various views of a reference frame.

FIG. 17 illustrates an isometric view of a reference frame guide andFIG. 18 illustrates the guide of FIG. 17 attached to the reference frameof FIG. 16A.

FIG. 19 illustrates the components of FIG. 18 being moved and positionfor attachment to the anatomy and FIG. 20 is an isometric viewillustrating said attachment.

FIG. 21 illustrates the removal of the guide frame and FIG. 22illustrates the remaining frame in position on the anatomy.

FIG. 23 illustrates another reference frame in position on the tibia.

FIGS. 24A, B and C illustrate a reference frame and its components.

FIG. 25 illustrates an implantation site on the tibia.

FIGS. 26A, 26B, and 26C illustrate another reference frame embodimenthaving a flexible linkage joining the components of the frame.

FIG. 26B1 a illustrates a flexible coupling in use about the upper andlower mount as shown in FIG. 26B. FIG. 26B1 b is an isometric view ofthe flexible coupling of FIG. 26B1 a.

FIG. 26B2 a illustrates a flexible coupling in use about the upper andlower mount of FIG. 26B. FIG. 26B2 b is an isometric view of theflexible coupling of FIG. 26B2 a.

FIGS. 27A and 27B illustrate two alternative reference frame surfaces.

FIG. 28 is an isometric view of an exemplary knee prosthesis near aschematically outlined distal femoral bone.

FIGS. 29A-29I and 30 illustrate the various views of an on tool trackingsystem and associated surgical tool in position for performance of atotal knee replacement OTT CAS procedure.

FIG. 31A is a flowchart representing an exemplary loop of a cyclic OTTCAS method (i.e looped or repeated in time).

FIG. 31B be is a flowchart providing additional details of the exemplaryprocessing steps performed using the method described in FIG. 31A.

FIG. 32 is a flow chart providing exemplary additional details of theprocessing steps used for determining a CAS processing mode.

FIG. 33 is a flowchart diagramming a number of factors considered asinputs for determining a CAS processing mode as well as a representativeoutputs.

FIG. 34 is a flowchart representing the exemplary OTT CAS mode adjustprocessing factors used to determine the process loads for a hover mode,a site approach mode and an active step mode.

FIG. 35 is a flowchart representing an exemplary OTT CAS processincluding the result of an OTT CAS process adaptation and the resultantmode algorithm and modified outputs thereof.

FIG. 36 is a flowchart representing an exemplary OTT CAS processincluding modification of any of the above described OTT CAS processesto include associated surgical tool operational characteristics,parameters or other data related to the use of an active element in anyOTT CAS process or procedure.

FIGS. 37A-44 relate to various alternative tactile feedback mechanismsalong with related kinematic responses and design criteria.

FIG. 37A illustrates a bent form that deflects to move an actuator inresponse to trigger force.

FIG. 37B illustrates a sliding trapezoid form that will deform andrestore its shape in response to trigger force.

FIG. 37C illustrates a rotating reader or encoder used to provide arotating response to the trigger force.

FIG. 37 D illustrates a frame moving in response to trigger force todepress a shaft into a base where the movement of the shaft may beregistered as an indication of trigger force.

FIG. 37 E illustrates a pinned element that may deflect to indicate anamount of trigger force.

FIGS. 38A and 38B illustrate a simple four bar mechanism, in a raisedand lowered, positions respectively that may be used to register triggerforce and displace a shaft.

FIGS. 39A, 39B and 39C each illustrate a scissor mechanism without aposition restoration element (39A), with a tension spring as a positionrestoration element (39B) and a compression spring as a positionrestoration element (39C).

FIG. 45 is an isometric view of a tactile feedback mechanism.

FIGS. 46A-46F illustrate various views of the components and operationof the mechanism of FIG. 45.

FIGS. 47 and 48 illustrate a side view of an on tool tracking devicemounted on a surgical instrument having a tool (here a saw) with thetactile feedback mechanism of FIG. 45 in position to interact with thetrigger of the surgical instrument. FIG. 47 illustrates the tactilefeedback mechanism in an expanded state configured to cover the triggerto prevent or attenuate manual pressing of the trigger and FIG. 48 showsthe tactile feedback mechanism collapsed to expose the trigger and allowmanual control.

FIGS. 49A-49B illustrate another alternative of a tactile feedbackmechanism in an open or expanded state (FIG. 49A) and a closed state(FIG. 49B).

FIGS. 49C-49E illustrate the various views of the internal mechanisms ofthe devices in FIGS. 49A and 49B.

FIG. 50 illustrates an embodiment of an OTT coupled for use with asurgical tool having an embodiment of the mechanism of FIGS. 49A and 49Bmounted for cooperation with the trigger of the surgical tool andconfigured to send and to receive trigger related signals with acomponent in the OTT.

FIG. 51 is a cut away view of an alternative embodiment of a scissormechanism utilizing two position restoration elements.

FIGS. 52A and 52B are front and rear isometric views respectively of anon tool tracking and navigation device (OTT) that includes a displaywith OTT housing coupled to a surgical tool having a trigger basedfeedback mechanism coupled to the OTT. The view also shows an exemplarycomputer system in communication with the OTT.

FIGS. 53-59B illustrate various OTT module and multiple cameraembodiments.

FIGS. 60-62B illustrate various OTT enabled sensor locations.

FIGS. 63, 64 and 65 are various flow charts related to various OTT CASmethods.

FIGS. 66A, 66B and 67 relate to various CAS displays.

DETAILED DESCRIPTION

The present invention is a system for performing computer assistedorthopedic surgery and novel tools for operating that system. Thepresent invention overcomes limitations of current computer assistedsurgery systems by optionally combining all elements of computerassisted surgery (tools, displays and tracking) into a single smartinstrument. The instrument does not rely on an external navigationsystem but the tool contains all the tracking equipment on the toolitself in a self-contained assembly. As a result, the overall system issignificantly less complicated, less intrusive to the surgeon and easyto integrate into existing practices in orthopedic surgery.

By way of overview, the system is comprised of principal subsystems. Thefirst is the tool itself, which is used to carry a standalone on tooltracking device or modified to contain the subsystems or elements of thesubsystems to provide On-Tool Tracking (OTT) functionality. Themodifications can be simple, such as an extended chassis to hold theadditional components, or complex, such as a modified power system topower the additional subsystems, and/or to stop or control the motorspeed or other actuators on the powered tool. The second subsystem isthe tracking subsystem, which comprises one or more trackers and one ormore tracking elements. The tracker can be a one, two (stereovision) ormore cameras that are sensitive to visible light or light from anotherwavelength. Alternatively, the tracker could be an electromagnetictracker or other non-camera based system. The tracking element iswhatever the tracker tracks. For example, where the tracker is aninfrared camera, the tracking element is an infrared LED, or a passivesurface reflective of infra-red light emitted from around the camera orelsewhere. Where the tracker is a pair of high-resolution camerassensitive to visible light, the tracking element could be the specificanatomy of a patient or marks made directly on the anatomy includingmarkers or reference frames. The subsystem can utilize one or moretrackers, mounted on the tool in various configurations, to track one ormore tracking elements. In one aspect, the tracker(s) (used to track thesensors required to track the tool, the patient and the other relevantobjects in order to perform an OTT CAS surgery) are located, at least inpart, on-board the surgical tool in a self-contained manner. Thenavigation system navigates when the tracking subsystem senses andcalculates the position (location and orientation/pose) of the trackingelement(s) relative to the tool.

The third subsystem is an OTT CAS computer system that contains anappropriate CAS planning software and programming to perform the OTT CASfunctions of the implementation of the surgical plan. The surgical plancan be produced and expressed through a variety of means but ultimatelycontains the locations, orientations, dimensions and other attributes ofthe resections (e.g. cuts, drill holes, volume of tissue to be removed),intended by the operator, in three-dimensional space. The system canalso contain a reference dataset from imaging of the patient's anatomy,such as a computed tomography image (dataset) of a patient's anatomy,and 2D or 3D virtual reconstructed models of the patient's anatomy, ormorphed models scaled to fit the patient anatomy as a point ofreference. The computer system compiles data from the tracking systemand the surgical plan to calculate the relative position of boundariesdefining the intended resections by the tool. In some configurations,the computer system can be a wholly separate component, in wirelesscommunication with the other components. In other configurations, thecomputer system is integrated into the other systems. Together, thetracking system and the computer system can determine if the surgeon'slocation, orientation and movement of the tool (the surgical path) willproduce the desired resection. It is important to note that the computersub system and the tracking sub system work together to establish thethree dimensional space of the surgical site. Elements necessary for thetracking sub-system to function can be located in the computersub-system or some intermediary mode of transmitting tracking data tothe computer sub-system.

The final subsystem is an indicator to provide the surgeon with OTT CASappropriate outputs related to his position, orientation and movement ofthe tool, as well as the intended resection, and the deviations (errors)between the two, within a real (or semi real) time OTT CAS step. Theindicator can be any variety of means to align/locate the surgical pathwith the intended resection: a panel of lights that sign directions tocorrect the surgeon, a speaker with audio instructions, a screen,touchscreen or iPhone or iPad or iPod like device (i.e., a so-called“smartphone”) on the OTT equipped tool displaying 3D representation ofthe tool and the patient with added guidance imagery or a digitalprojection (e.g., by a picoprojector) onto the patient's anatomy of theappropriate location of a resection. The indicator serves to provide anappropriate OTT CAS output to guide the surgeon to make the rightresection based on real time (or semi-real time) information.

Looking now to the specific subsystems:

A surgical suite for computer assisted surgery includes a first computerfor pre-operative planning use. For example, pre-operative analysis ofthe patient and selection of various elements and planned alignment ofthe implant on the modeled anatomy may be performed on the firstcomputer. The suite may also include a second computer, referred to asthe OR computer, which is used during a procedure to assist the surgeonand/or control one or more surgical instruments. In addition the suitemay include a computer (standalone or collaborating with anothercomputer) mounted on the surgical instrument via an embodiment of an ontool tracking system. Finally, one or more computers are used asdedicated drivers for the communication and medium stage data processingfunctions interfaced to the cutting instrument tracking system, motorcontrol system, or projection or display system. The first computer isprovided in the present instance, but may be omitted in someconfigurations because the functions of the computer are alsoimplemented on the OR computer, which can be a standalone. Moreover thewhole ‘pre-surgical planning’ may eventually happen instantaneouslyinside the OR using primarily the OR computer in conjunction with anOTT. Nevertheless, if desired for particular applications, the firstcomputer may be used. The pre-surgical planning and procedure can alsobe aided by data or active guidance from online web-links. As usedherein, the term CAS system or CAS computer refers to those computers orelectronic components as provided in any of these combinations toperform CAS function. Furthermore, the micro-processing unit of thesystem can reside in the on tool tracking instrument. In such aconfiguration, the computations and user interface can be performedwithin a computer borne on the surgical tool being used, or incollaboration with the main system computer by wired or wirelesscommunications, and some of which can be done through the sub-system“driver” computers. In collaboration with the main OTT CAS computer bydirect wireless communication or indirect through the intermediarydriver computers, such system performs error analysis of location of thecutting instrument relative to the ideal cut to be performed, anddisplays corrective actions and other information on a screen providedas part of the on tool tracker alone or in any combination with anoutput provided by one or more projectors provided with the OTT for thatpurpose.

As a result, a surgical suite for OTT CAS may include atracking/navigation system that allows tracking in real time of theposition and orientation in space of several elements, including: (a)the patient's structures, such as the bone or other tissue; (b) thesurgical tool, such as the bone saw and/or OTT, which carries the OTTand is controlled by the surgeon based on information from the ORcomputer or (c) surgeon/assistance specific tools, such as a navigatedpointer, registration tools, or other objects as desired. The ORcomputer or an OTT may also perform some control on the instrument.Based on the location and orientation (pose) of the tool and feedbackfrom an OTT, the system or CAS computer is able to vary the speed of thesurgical tool as well as turn the tool off to prevent potential damage.Additionally, the CAS computer may provide variable feedback to a user.The surgical instrument shown in the accompanying description is asurgical saw. It is to be appreciated that many others instruments canbe controlled and/or navigated as described herein, such as a drill,reamer, burr, file, broach, scalpel, stylus, or other instrument.Therefore in the following discussion, the OTT enabled CAS system is notlimited to the particular tool described, but has application to a widevariety of instruments and procedures.

As discussed further below, one exemplary use of the surgical suiteincorporates the use of a virtual model of the portion of the patientupon which a procedure is to be performed. Specifically, prior to aprocedure, a three dimensional model of the relevant portion of thepatient is reconstructed using CT scans, MRI scans or other techniques.Prior to surgery, the surgeon may view and manipulate the patient modelto evaluate the strategy for proceeding with the actual procedure.

One potential methodology uses the patient model as a navigation deviceduring a procedure. For instance, prior to a procedure, the surgeon mayanalyze the virtual model of a portion of the patient and map out thetissue to be resected during a procedure. The model is then used toguide the surgeon during the actual procedure. Specifically, during theprocedure, the on tool tracking device monitors the progress of theprocedure. As a result of the OTT CAS processes performed, theprogress/results are displayed in real time on the OR computer or on anOTT monitor (e.g. onboard LCD screen) so that the surgeon can see theprogress relative to the patient model. Importantly, the surgeon is alsoprovided an OTT projector to provide real type feedback based on OTT CASprocessing steps (described in greater detail below).

To provide navigation assistance during an OTT CAS procedure, an on tooltracking device monitors the position of the associated surgical toolwithin the surgical field. The OTT CAS system may use none, or one ormore reference frames including one or more positions sensors or one ormore fiducial markers depending upon the requirements of the OTT CASprocedure being undertaken. Any of the above described markers may beutilized in an active or passive configuration. Markers may, optionally,be wired or wireless sensors that are in communication with the system.An active marker emits a signal that is received by the OTT device. Insome configurations, the passive markers are (naturally wireless)markers that need not be electrically connected to the OTT CAS system.In general, a passive marker reflects infrared light back to anappropriate sensor on the OTT device. When using passive markers, thesurgical field of view is exposed to infrared light that is thenreflected back to and received by the OTT, from which the data locationsof the passive markers are determined by the OTT CAS, and from such datathe location and orientation of the surgical site, and other instrumentsare computed relative to the OTT and to each other. Some embodiments ofan OTT device may be provided with an infrared transmission device andan infrared receiver. The OTT receives emitted light from the activemarkers and reflected light from the passive markers along with othervisual field information reaching the OTT. The OTT CAS system performscalculations and triangulates the three dimensional position andorientation of the tool based on the vision processing of the imagesincluding the position of the markers along with other imaginginformation in the surgical field. Embodiments of the on tool trackingdevice are operable to detect the position and orientation of theOTT-enabled tool relative to three orthogonal axes. In this way, usinginformation from the OTT device, the OTT CAS system determines thelocation and orientation of the tool, and then uses that information todetermine OTT CAS processing modes and produce appropriate OTT CASoutputs for the user.

As is typical in navigation and other CAS systems, a series of points orsurfaces are used to register or correlate the position of the patient'sanatomy with the virtual model of the patient. To gather thisinformation, a navigated pointer is used to acquire points at ananatomical landmark or a set of points on a surface within the patient'sanatomy. A process referred to as morphing (or kinematic registration)may alternatively be used to register the patient to an approximate(scaled) virtual model of the patient taken from an atlas or databaseand not originating from actual imaging of that particular patient.During such a process, the surgeon digitizes parts of the patient andsome strategic anatomical landmarks. The OTT CAS computer analyzes thedata and identifies common anatomical features to thereby identify thelocation of points on the patient that correspond to particular pointson the virtual model.

Accordingly, as set forth above, the on tool tracking device visuallymonitors the position of several items in real time, including: theposition of the associated surgical tool, the position of the patientand the position of items used during a procedure, such as one or morereference frames or one or more markers. Accordingly, the OTT CAScomputer processes the OTT CAS data regarding the position of theassociated surgical tool, visual field information in OTT image data,the data regarding the position of the patient, and the data regardingthe model of the patient. This result of OTT CAS computer processesprovide dynamic, real time interactive position and orientation feedbackinformation, which can be viewed by the surgeon on a monitor provided bythe OTT device (if provided) or as a displayed output of an OTTprojector. Further still, as previously described, prior to a procedure,the surgeon may analyze the patient model and identify the tissue thatis to be resected as well as plan for or indicate desired OTT CAS modefor use during an OTT CAS step or during a CAS procedure. Thisinformation can then be used during the procedure to guide the surgeonusing dynamically adjusted outputs based on the mode of CAS processingand other factors.

FIG. 1 is an isometric view of an on tool tracking device (OTT) 100arranged for tracking and providing guidance during computer aidedsurgery using the surgical instrument 50. The OTT 100 has a housing 105that includes a pair of cameras 115, in an opening for projector output110. The OTT 100 and also as a housing 105 with a surface 120 adaptedand configured to mate with the surgical instrument 50. The surgicalinstrument 50 includes a trigger 52 for operating a tool 54 having anactive element 56. An illustrative embodiment of FIG. 1 the tool 54 is asaw and the active element 56 is the serrated edge of a saw blade at thedistal end thereof.

FIG. 2 is an isometric view of an on tool tracking device (OTT) 200 andarranged for tracking and providing guidance during computer aidedsurgery using the surgical instrument 50. The OTT 200 has a housing 205that includes a pair of cameras 215, in an opening for projector output210. The OTT 200 and also as a housing 205 with a surface 220 adaptedand configured to mate with the surgical instrument 50. The surgicalinstrument 50 includes a trigger 52 for operating a tool 54 having anactive element 56. An illustrative embodiment of FIG. 2 the tool 54 is asaw and the active element 56 is the serrated edge of the distal endthereof.

FIGS. 3 and 4 are isometric views of the on tool tracking devices ofFIGS. 1 and 2 with the top cover of the housings removed. In the view ofFIG. 3, the interior of the housing 105 exposed in shows the placementof the processing circuits 130, projector 125 and cameras 115. Theprojector 125 is illustrated in this embodiment in the position above aplane containing the cameras 115, but tilted to make the output of theprojector 125 more symmetrically above and below the plane of thecameras 110. The projector can be tilted further or less vertically andsome horizontally if needed in special situations, to optimize the imageit projects with respects to various criteria such as occlusion (e.g.,by the saw blade in FIGS. 3 and 4, or drill bits) or specifics of thenature, shape, reflection and other aspects of the anatomy or surfaceupon which the image is projected onto. In the view of FIG. 4, theexposed interior of the housing 205 shows the placement of theprocessing circuits 230, projector 225 and cameras 215. The output 210of the projector 225 is illustrated in this embodiment in a positionabove that, and at an acute angle with a plane containing the cameras215.

FIGS. 5, 6, and 7 represent one top down, and two isometric views of theon tool tracker 200. In the top down view of the on tool tracker shownin FIG. 4 the orientation and arrangement of the electronic componentsis clearly visible. As a result of the type of projector 225 used inthis configuration, the projector has been positioned within the housing205 at an angle and, as shown in FIG. 6 on a slightly inclined surface.In one embodiment, either or both of the cameras or the projector of anon tool tracking device may be positioned in any orientation and theresult of that orientation to the operation of the respective device isthen compensated for in other ways as described herein. In this way,various different OTT electronic circuits and component designs arepossible since the slight physical misalignments may be adjusted forusing software techniques as described herein. FIG. 7 illustrates anisometric view of the electronic components of the on tool tracker 200separated from the housing 205. This figure illustrates one embodimentof a quote one piece” OTT electronics package having cameras 215,projector 225 and associated system and processing electronics 230 on asingle board 235 for placement within the housing 205.

FIGS. 8A, 8B, 9 and 10 all illustrate the result on camera field of viewfor various angle orientations for the cameras included within an ontool tracking device. The cameras 115 in FIG. 8A are oriented in nearlyparallel arrangement with regard to one another and the axis of thesurgical tool 54. After accounting for blockage caused by othercomponents, this configuration provides a camera field of view rangingfrom about 70 mm to about 200 mm. In other embodiments, the camerasystems of an exemplary OTT device may operate in a camera field of viewranging from about 50 mm to about 250 mm. It is to be appreciated thatthe camera field of view may be physically or electronically altereddepending upon the desired field of view needed for the particularcomputer aided surgery procedure that the OTT device will be used toperform.

In contrast to the nearly parallel arrangement of the cameras in FIG.8A, FIGS. 8B, 9 and 10 each demonstrate the result of different cameratilt angles and the resulting alteration of the camera field of view.The relationship of OTT camera positioning and tilt angle and theirrelationship to the angle of view, minimum object distance and maximumobject length are better appreciated with reference to FIGS. 11A, 11B,11C and 11D. FIG. 11A illustrates the geometric set up and formula formaking the calculations used to produce the chart in FIG. 11B thatrelates tilt angle in degrees to a number of vision field factors. Thedata from this chart related to tilt angle is reproduced in the graphsshown in FIGS. 11C and 11D. The optical field information presented inthese figures is useful in the design and optimization of camerapositioning in some of the various embodiments of the OTT devicesdescribed herein.

Additional aspects of the projector used with the various OTTembodiments may be appreciated for reference to FIGS. 12A, 12 B, 13A,13B, and 13C. The impact on projector output based upon projectorpositioning within the OTT housing is demonstrated by a comparisonbetween FIG. 12A and FIG. 12B. The projector 125 appears to be in anearly planar relationship relative to the tool 54 as shown in bothFIGS. 12A and 13A. However, notice how a portion of the projector output126 extends beyond and below the tool (in this case saw blade) distalend 56. In contrast, the projector 225 is positioned at an acute anglein relation to the tool 54. Additionally, the projector 210 output isoff to one side when compared to its relative position between thecameras 215. However, the projector output 226 is mostly above the blade54 and crosses only at the distal end 56. Additional aspects of theprojector output 226 are apparent upon review of the views in FIGS. 13Aand 13B. It is to be appreciated that the projector outputs, projectorsize and orientations described in these embodiments is not limiting toall OTT device embodiments. A suitable OTT projector may be configuredin a number of satisfactory ways and placement within the OTT housing,and may be adjusted based on package size of a desired projector. As isclearly illustrated by the sample outputs of the projector 225, manydifferent projector sizes, orientations and angular relationships may beused and still be effectively operated to meet the projectorrequirements of the OTT CAS processing system. In other words, a widevariety of projector types, output locations and packaging may be usedand still remain within the various embodiments of the OTT devicesdescribed herein.

Embodiments of the OTT device of the present invention are provided witha variety of imaging, projector and electronic components depending uponthe specific operational characteristics desired for a particular OTTCAS system. The illustrative embodiments that follow are provided inorder that the wide variety of characteristics and design factors may beappreciated for this part of the OTT CAS system.

FIG. 14A illustrates a schematic of an embodiment of an OTT device. Inthis illustrated embodiment, there is provided

-   -   Camera/dsp/processing (eg. NaturalPoint Optitrak SL-V120range)    -   Computer: PC—Windows 2000/XP/Vista/7; 1.5 GHz Processor; 256 MB        of RAM; 5 MB of free hard disk space; USB 2.0 Hi-Speed port        (minimum, faster is better)    -   COM: Wireless Communication (eg. USB Port Replicator with        Wireless USB support)    -   Projector: (Laser Pico Projector type)        that are arranged within the OTT housing as shown in the view.        This embodiment makes use of what is known as ‘smart        cameras’—cameras that have the capability of performing        localized image processing. This processing can be programmable        usually through Field Programmable Gate Arrays (FPGAs). The        configuration of the components in this specific embodiment are        utilized to provide image processing that occurs both on the OTT        devices and on a OTT CAS computer. For example, DSP on the OTT        device detects and processes marker data before transferring it        to the OTT CAS computer. The configuration greatly reduces        processing power required on the host computer while also        minimizing the data needed to transmit. It is to be appreciated        that the schematic view, while useful primarily to show the type        of imaging, data processing and general computer processing        capabilities of a particular OTT device or as between an OTT        device and a OTT CAS computer, or as between an OTT device and        one or more intermediary device driver computers, this view may        not reflect the actual orientation, spacing and/or alignment        between specific components. Electronic communications        capabilities (COM) are provided via wired connection or any        suitable wireless data transfer mode from and to a computer that        is adapted and configured for use with OTT CAS processes,        algorithms and modes described herein. The type, variety,        amount, and quality of the processing data exchange between the        OTT device and an OTT CAS computer (if used) will vary depending        upon the specific parameters and considerations of a particular        OTT CAS procedure, mode or system utilized.

FIG. 14B illustrates a schematic of an embodiment of an OTT device. Inthis illustrated embodiment, there is provided

-   -   Camera: Analog camera wired or wireless; eg FPV wireless camera    -   DSP: uCFG Microcontroller Frame Grabber. This is connected to        the PC PCI bus and becomes part of the PC.    -   Computer: Computer: PC—Windows 2000/XP/Vista/7; 1.5 GHz        Processor; 256 MB of RAM; 5 MB of free hard disk space; USB 2.0        Hi-Speed port (minimum, faster is better)    -   COM: Hardwiring or Analog wireless transmitter    -   Projector: Microvision's SHOWWX Laser Pico Projector        that are arranged within the OTT housing as shown in the view.        The configuration of the components in this specific embodiment        are utilized to provide use of low cost commodity cameras where        no image processing for tracking is performed onboard the OTT        and the image signal is captured by a dedicated frame grabber        that is part of the PC. The frame grabber accepts the captured        image and deposits it into PC memory without any overhead        processing by the PC. This embodiment results in a smaller,        lighter and lower cost OTT device.

It is to be appreciated that the schematic view, while useful primarilyto show the type of imaging, data processing and general computerprocessing capabilities of a particular OTT device or as between an OTTdevice and a OTT CAS computer or via one or more intermediary devicedriver computers, this view may not reflect the actual orientation,spacing and/or alignment between specific components. Electroniccommunications capabilities (COM) are provided via wired connection orany suitable wireless data transfer mode from and to a computer that isadapted and configured for use with OTT CAS processes, algorithms andmodes described herein. The type, variety, amount, and quality of theprocessing data exchange between the OTT device and an OTT CAS computer(if used) will vary depending upon the specific parameters andconsiderations of a particular OTT CAS procedure, mode or systemutilized.

FIG. 15A illustrates a schematic of an embodiment of an OTT device. Thisembodiment utilizes commodity USB cameras with incorporated electroniccircuitry that captures the image from the camera and conditions it tobe USB compatible. This output is compressed and then transmittedthrough wires or wirelessly without further tracking related processing.

In this illustrated embodiment, there is provided

-   -   Camera: (e.g., miniature webcam)    -   Computer: (e.g., Dell Precision RS500 Rack Workstation)    -   COM: [e.g., Carambola 8 devices Core, or DTW-200D (CDMA2000 1X)        and DTW-500D (EVDO Rev A)]    -   Miniature Projector: (e.g., Microvision's SHOWWX Laser Pico        Projector)        that are arranged as shown in the view. The configuration of the        components in this specific embodiment are utilized to provide a        modular solution for providing the electronic OTT components.        This embodiment uses commodity low cost cameras and allows the        cameras to be used in a modular form where they can be changed        or upgraded to reflect advances in technology without disrupting        the OTT or the ground based systems.

There is no need to use an on-tool DSP if the OTT CAS or intermediarydriver computer is optimized for DSP. This embodiment makes it possibleto use any of the commercially available image processing libraries. Forexample, modern image processing software routines from open source orcommercial libraries take only about lms to process blobs (bonereference frame LEDs) and compute their centroids. Images can thereforebe sent directly from the OTT tool to the OTT CAS Computer to beprocessed. It is important that the COM will need to be selected tohandle higher bandwidth when compared to other embodiments. Similarly,the intermediary driver or OTT CAS Computer will need to be selected tohandle more burdensome computation.

It is to be appreciated that the schematic view, while useful primarilyto show the type of imaging, data processing and general computerprocessing capabilities of a particular OTT device or as between an OTTdevice and an intermediary driver or an OTT CAS computer, this view maynot reflect the actual orientation, spacing and/or alignment betweenspecific components. Electronic communications capabilities (COM) areprovided via wired connection or any suitable wireless data transfermode from and to a computer that is adapted and configured for use withOTT CAS processes, algorithms and modes described herein. The type,variety, amount, and quality of the processing data exchange between theOTT device and an intermediary driver (if used) or OTT CAS computer (ifused) will vary depending upon the specific parameters andconsiderations of a particular OTT CAS procedure, mode or systemutilized.

FIG. 15B illustrates a schematic of an embodiment of an OTT device. Inthis illustrated embodiment, there is provided

-   -   Camera: Smart camera as in FIG. 15A or USB camera as in FIG. 15C    -   Inertia Sensors: (e.g., Bosch SMB380, Freescale PMMA7660, Kionix        KXSD9)    -   Onboard processor: (e.g., ARM processor)    -   Computer: [e.g., PC—Windows 2000/XP/Vista/7; 1.5 GHz Processor;        256 MB of RAM; 5 MB of free hard disk space; USB 2.0 or USB 3.0        Hi-Speed port (minimum, faster is better)]    -   COM: (Standard IEEE 802.11 communications protocol or similar        protocol for communication between the OTT borne processor and        the ground station intermediary driver PC or OTT CAS PC.    -   Projector: (e.g., Microvision's SHOWWX Laser Pico Projector)        that are arranged as shown in the view. The configuration of the        components in this specific embodiment are utilized to provide        an embodiment that performs complex processing onboard the OTT        device to accomplish most of the body tracking as needed for        purposes of OTT CAS procedures. The device is a complete        stand-alone tracking device. The OTT device further contains one        or more inertia sensors. DSP involves the use of Inertia sensors        to predict the location of the fiducials in the ‘next frame’. As        a result, the computational burden on the DSP on the OTT device        is minimized.

It is to be appreciated that the schematic view, while useful primarilyto show the type of imaging, data processing and general computerprocessing capabilities of a particular OTT device or as between an OTTdevice and an intermediary driver or OTT CAS computer, this view may notreflect the actual orientation, spacing and/or alignment betweenspecific components. Electronic communications capabilities (COM) areprovided via wired connection or any suitable wireless data transfermode from and to a computer that is adapted and configured for use withOTT CAS processes, algorithms and modes described herein. The type,variety, amount, and quality of the processing data exchange between theOTT device and directly to an OTT CAS computer (if used) or via anintermediary driver computer will vary depending upon the specificparameters and considerations of a particular OTT CAS procedure, mode orsystem utilized.

In addition to the above described details and specific embodiments, itis to be appreciated that alternative embodiments of an OTT device mayhave electronic components including components with processingcapabilities as well as software and firmware and electronicinstructions to provide one or more of the following exemplary types ofOTT CAS data in accordance with the OTT CAS processing methods, modesand algorithms described herein:

-   -   Receive and process visual and IR spectrum image data    -   Determining coordinates of the centroid of each of the markers        within image frame    -   Determining the sizes of all markers within an image frame    -   Reporting the size and the coordinates of one or more fiducials    -   Sub-pixel analysis to determine the location of the centroid        within an image frame, a marker placement or selected marker        placements    -   Variable and controllable frame rate from 10 to 60 frames per        second based on input from central computer or internal        instructions or in response to an OTT CAS processing mode        adaptation

The inventive on tool tracking devices 100/200 illustrated and describedin FIGS. 1-15B and FIGS. 47-52B may also include, for examples, one ormore additional cameras, different types of camera functionality, aswell as sensors that may be employed by an OTT CAS system as describedherein and in FIGS. 31A-36, 63, 64 and 65. Various different OTTconfigurations will be described with reference to FIGS. 53-63A and 63B.

FIG. 53 is an isometric view of the on tool tracking device 100 mountedon the surgical tool 50. The embodiment of the on tool tracking device100 illustrated in FIG. 53 a modified housing 105 and on-boardelectronics to include a pair of near field stereoscopic cameras 245 a,245 b. In this embodiment the cameras 245 a, 245 b are mounted adjacentto the projector output or opening 110 near the top of the OTT housing105. As described herein, the cameras 115 may be used to provide a widefield of view. The cameras 115 are mounted at the midpoint of thehousing 105. The wide view stereoscopic cameras 115 are just above theplane that contains the surgical tool 54 that is being tracked by theOTT CAS system. In one aspect, the cameras or wide view cameras 115 areon opposite sides of the tool 54 under OTT CAS guidance. The OTT CASsystem operation is similar to that described below in FIGS. 31A to 36and FIGS. 63, 65 and 65 with the use of the additional camera inputs anddata available for OTT CAS methods and techniques. The OTT CAS systemand methods of performing freehand OTT CAS may be adapted to receiveinputs from one or sets of cameras 115, 245 a, 245 b or from one or moreof cameras 115, 245 a, 245 b in any combination. Furthermore, any cameraof those illustrated may be used for tracking, display, measurement orguidance alone or in combination with the projector 225 in one or modesof operation under control of the OTT CAS system described herein.

FIG. 54 is an isometric view of the on tool tracking device 200 mountedon the surgical tool 50. As described herein, the cameras 215 aremounted at the midpoint of the housing 205 used to provide a wide fieldof view. In this alternative embodiment of the on tool tracking deviceillustrated in FIG. 54, the housing 205 and on-board electronics aremodified to include a pair of near field stereoscopic cameras 245 a, 245b as in FIG. 53 along with additional cameras 317 a, 317 b, 319 a, and319 b. The additional cameras may provide, for example, an additionalwide field view (i.e., wider than that provide by cameras 215) or beconfigured as IR cameras. As with FIG. 53 the cameras 245 a, 245 b aremounted adjacent to the projector output or opening 110 near the top ofthe OTT housing 205. Cameras 319 a and 319 b are shown mounted adjacentto the projector output or opening 210 near the top of the OTT housing205. The wide view stereoscopic cameras 215 are just above the planethat contains the surgical tool 54 that is being tracked by the OTT CASsystem. Additional cameras 317 a, 317 b are provided between the cameras245 a, 245 b and the cameras 215. In one aspect, the cameras or wideview cameras 215 are on opposite sides of the tool 54 under OTT CASguidance. The OTT CAS system operation is similar to that describedbelow in FIGS. 31A to 36 and FIGS. 63, 65 and 65 with the use of theadditional camera inputs and data available for OTT CAS methods andtechniques. The OTT CAS system and methods of performing freehand OTTCAS may be adapted to receive inputs from one or sets of cameras 215,245 a, 245 b, 317 a, 317 b, 319 a or 319 b or from one or more ofcameras 215, 245 a, 245 b, 317 a, 317 b, 319 a or 319 b in anycombination. Furthermore, any camera of those illustrated may be usedfor tracking, display, measurement or guidance alone or in combinationwith the projector 225 in one or modes of operation under direct orindirect (via intermediary driver computer) control of the OTT CASsystem described herein.

FIG. 55 is an isometric view of the on tool tracking device 100 mountedon the surgical tool 50. The embodiment of the on tool tracking device100 illustrated in FIG. 55 has a modified housing 105 and on-boardelectronics to include a single, centrally located camera 321 locatedabove the projector output 110. In this embodiment the camera 321 ismounted adjacent to the projector output or opening 110 build into thetop of the OTT housing 105. As described herein, the camera 321 may beused to provide a variety of different fields of view either throughmechanical or electronic lens control alone or in combination withsoftware based imaging processing. As illustrated, the camera 321 ismounted at or near the central axis of the tool 54 with a clear view ofthe active element 56 or other tracking point on the tool 50. Thestereoscopic cameras 115 are also shown just above the plane thatcontains the surgical tool 54 that is being tracked by the OTT CASsystem. In one aspect, the cameras 115 are on opposite sides of the tool54 under OTT CAS guidance. The OTT CAS system operation is similar tothat described below in FIGS. 31A to 36 and FIGS. 63, 65 and 65 with theuse of the additional camera input and data available for OTT CASmethods and techniques. The OTT CAS system and methods of performingfreehand OTT CAS may be adapted to receive inputs from one or sets ofcameras 115 or 321 or from one or more of cameras 115 or 321 in anycombination. Furthermore, any camera of those illustrated may be usedfor tracking, display, measurement or guidance alone or in combinationwith the projector 225 in one or more modes of operation under direct orindirect control of the OTT CAS system described herein.

FIG. 56 is an isometric view of the on tool tracking device 200 mountedon the surgical tool 50. This OTT device embodiment is similar to thatof FIG. 54 with an addition single camera provided as in FIG. 55. Incontrast to FIG. 55, the single camera 323 in FIG. 56 is provided belowthe tool 53 and active element 56 being tracked under an OTT CAS system.One advantage of the location of camera 323 is that some tools 54—suchas the illustrated saw—may block portions of the views available toother camera. In those instances, the input from camera 323 may be usedto augment other imaging inputs provided to the OTT CAS system.Additionally, the camera 323 may be particularly useful in monitoringone or more reference frames or markers used as part of the OTT CASguidance of the attached surgical tool 50. As described herein, thecameras 215 are mounted at the midpoint of the housing 205 used toprovide a wide field of view. In this embodiment the camera 323 ismounted in a forward projection of the housing 205 below the tool 54. Asdescribed herein, the camera 323 may be used to provide a variety ofdifferent fields of view either through mechanical or electronic lenscontrol alone or in combination with software based imaging processing.As illustrated, the camera 323 is mounted at or near the central axis ofthe tool 54 with a clear view of the underside of the active element 56or other tracking point on the tool 50. In this alternative embodimentof the on tool tracking device illustrated in FIG. 54, the housing 205and on-board electronics are modified to include the various cameras ofFIG. 54 along with the single camera 323. The OTT CAS system operationis similar to that described above with reference to FIG. 54 as well asbelow in FIGS. 31A to 36 and FIGS. 63, 65 and 65 with the use of theadditional camera inputs and data available for OTT CAS methods andtechniques. The OTT CAS system and methods of performing freehand OTTCAS may be adapted to receive inputs from one or sets of cameras 215,245 a, 245 b, 317 a, 317 b, 319 a, 319 b or 323 or, from one or more ofcameras 215, 245 a, 245 b, 317 a, 317 b, 319 a, 319 b or 323 in anycombination. Furthermore, any camera of those illustrated may be usedfor tracking, display, measurement or guidance alone or in combinationwith the projector 225 in one or modes of operation under control of theOTT CAS system described herein. It is to be appreciated that the singlecameras as shown in FIGS. 55 and 56 may be combined into an OTT deviceas illustrated in FIG. 55 or in combination with other OTT deviceembodiments.

FIG. 57A is an isometric view of the on tool tracking device 100 mountedon the surgical tool 50. The embodiment of the on tool tracking device100 illustrated in FIG. 57 has a modified housing 105 and on-boardelectronics to include an additional pair of cameras 241 a, 241 blocated about the same aspect as cameras 115 and below the projectoroutput 110. In this embodiment the cameras 241 a, b are mounted in theOTT housing 105 as with cameras 115. As described herein, the cameras115, 241 a, 241 b may be used to provide a variety of different fieldsof view either through mechanical or electronic lens control alone or incombination with software based imaging processing. As illustrated inFIG. 57B the cameras may by be used to provide different fields of vieweither by angling the cameras or by having the cameras 115 241 a, 241 bmounted on a movable stage that provides for altering the direction ofcamera orientation. FIG. 57B illustrates the embodiment where thecameras 115 are directed inwardly towards the central axis of the toolwhile the cameras 241 a, 241 b are directed outward of the central axis.The cameras may obtain the orientations of FIG. 57B by fixed or movablestages. The cameras in FIG. 57A, 57B are also shown just above the planethat contains the surgical tool 54 that is being tracked by the OTT CASsystem. In one aspect, one camera of each pair of cameras is provided onopposite sides of the tool 54 under OTT CAS guidance. The OTT CAS systemoperation is similar to that described below in FIGS. 31A to 36 andFIGS. 63, 65 and 65 with the use of the additional camera input and dataavailable for OTT CAS methods and techniques. The OTT CAS system andmethods of performing freehand OTT CAS may be adapted to receive inputsfrom one or sets of cameras 115 or 241 a, 241 b or from one or more ofcameras 115 or 241 a, 241 b in any combination. Furthermore, any cameraof those illustrated may be used for tracking, display, measurement orguidance alone or in combination with the projector 225 in one or modesof operation under control of the OTT CAS system described herein.

FIG. 58 illustrates another alternative embodiment of camera variationfor the configuration illustrated in FIGS. 57A and 57B. In onealternative aspect, the cameras of FIG. 57A may be adjusted—via softwareor other suitable imaging processes—to provide the view of viewillustrated in FIG. 58. In this embodiment, two pairs of cameras areprovided as with the embodiment of FIG. 57A. In this embodiment of thecamera of the OTT system, the camera angles A do not overlap as shown.The A angles are used to enhance the sides of the tool 54. In the imageprocessing system the various views are synthesized into a unified viewby the image processing system of the CAS tracking and guidance system.FIG. 58 illustrates the upper cameras (241 a, 241 b or A cameras) with anarrow and non-overlapping field of view within the surgical field. Thelower cameras (115 or B cameras) have a wider and overlapping field ofview. In this embodiment, the image tracking system is able to use thewider overlapping field of view and the narrow focused fields of view inorder to provide a variety of different tracking schemes by synthesizingand obtaining information from the various camera views that areprovided. The OTT CAS system operation is similar to that describedbelow in FIGS. 31A to 36 and FIGS. 63, 65 and 65 with the use of theadditional camera input and data available for OTT CAS methods andtechniques. The OTT CAS system and methods of performing freehand OTTCAS may be adapted to receive inputs from one or sets of cameras 115 or241 a, 241 b or from one or more of cameras 115 or 241 a, 241 b in anycombination. Furthermore, any camera of those illustrated may be usedfor tracking, display, measurement or guidance alone or in combinationwith the projector 225 in one or modes of operation under control of theOTT CAS system described herein.

FIG. 59A is an isometric view of the on tool tracking device 200 mountedon the surgical tool 50. This OTT device embodiment is similar to thatof FIG. 54 with a moveable camera stage 244 in place of camera pair 315a, 315 b and without camera pair 319 a, 319 b. In this alternativeembodiment of the on tool tracking device illustrated in FIG. 59A, thehousing 205 and on-board electronics are modified to include a moveablecamera stage 244 and included camera pair 247 a, 247 b. As in FIG. 54,the embodiment of FIG. 59A also includes cameras 215, 317 a, and 317 b.The additional cameras may provide, for example, an additional field orvariable fields of view through OTT CAS system controlled operation ofthe stage 244. The stage 244 is shown mounted adjacent to the projectoroutput or opening 210 near the top of the OTT housing 205. The stage 244is provided with motors, a stage or other controlled movement devicepermitting the spacing between and/or angulation and/or focus of thecameras 247 a, 247 b to change. As best seen in FIG. 59B the cameras 247a, 247 b may move from a wide angle position (“a” positions) a mid-rangeposition (“b” positions) or a narrow range position (“c” position).

In addition or alternatively, the camera motion and selection of viewalong with the control of the camera motors, stage or other movementdevice are, in some embodiments, controlled based on user selectedinputs such as a pre-set camera view in a smart views system. In stillanother alternative, the position or orientation of a camera or camerastage or motion device may vary automatically based upon the operationsof an embodiment of the CAS hover control system described herein. Byutilizing the camera movement capabilities of this embodiment, the imagetracking system is also able to use a camera motor controller to obtainwider, mid-range or narrow field imaging as desired based on other CAShover system parameters and instructions. As such, the moving cameracapabilities of this embodiment of an OTT system provides a variety ofdifferent tracking schemes by synthesizing and obtaining informationfrom the various camera views that are provided by the camera motion.The OTT CAS system operation is similar to that described below in FIGS.31A to 36 and FIGS. 63, 65 and 65 with the use of the additional camerainputs and data available for OTT CAS methods and techniques as well asthe ability for the OTT CAs system to control the movement of cameras247 a, 247 b depending upon OTT CAS techniques and methods describedbelow. The OTT CAS system and methods of performing freehand OTT CAS maybe adapted to receive inputs from one or sets of cameras 215, 247 a, 247b, 317 a, or 317 b or from one or more of cameras 215, 247 a, 247 b, 317a, or 317 b in any combination. Furthermore, any camera of thoseillustrated may be used for tracking, display, measurement or guidancealone or in combination with the projector 225 in one or modes ofoperation under control of the OTT CAS system described herein.

In still further alternative aspects, it is to be appreciated that anyof the OTT device embodiments described herein may, in addition tohaving multiple cameras or sets of cameras, may provide each camera withfilters via hardware and/or software so that each camera may be used ineither or both of the visible spectrum and the infrared spectrum. Insuch case, the two pairs of cameras can be thought as four set ofcameras since in one sense the camera operates in the visible field andthen those same cameras are operated via filters in the infrared field.

In still further alternative aspects, the OTT device embodimentsdescribed herein may, in addition to having multiple cameras or sets ofcameras, may utilize any one or more of the onboard cameras to captureimages for the purpose of recording and zooming while recording acertain aspect of the procedure for documentation, training orassessment purposes. In still another aspect, there is provided on anOTT module in software or firmware instructions a rolling recording loopof a preset time duration. The time duration could be any lengt of timeas related to a complete OTT CAS procedure, step or portion of a step orplanning or registration as related to a OTT CAS procedure or use of anOTT CAS device. There may be storage provided directly on the OTT CAS oron a related computer system. In one aspect, an OTT CAS module orelectronics device includes a memory card slot or access to permitrecording/storing the camera and/or projector outputs along with all ora portion of a OTT CAS surgical plan or images used in an OTT CAS plan.Still further, the video data and image storage may be on the OTT eithera USB or other port or there is just a memory card as is common withhandheld video cameras. The feed from the OTT camera(s) is recordedeither on command, always on or done in response to a user or systeminput such as a mouse click, touch screen input, voice commant and thelike. Imaging data may be stored on the OTT itself or a device oranother computer. In one example, the OTT CAS image data referenced hereis stored, for example, on an intermediary driver computer. In stillanother aspect, the recording mentioned herein is started manually froma remotely sent command to the OTT from the master CAS computer, or,optinally from a touch screen command of the LCD screen onboard the OTTdevice. The commands can be “start video recording”, stop videorecording”, “capture single image” etc. The recorded data or storedimages can be stored locally on the OTT, and/or immediately or laterrelayed to the intermediary driver computer or to the master CAScomputer to be associated with the surgical case file.

FIGS. 60, 61, 62A and 62B provide various alternative views of the OTTdevice electronics package illustrated and described with reference toFIGS. 5, 6 and 7. The various views of FIGS. 60, 61, 62A and 62Billustrate the wide variety of locations and sensor types thatoptionally may be incorporated into the various embodiments of the OTTdevice as well as providing further inputs, processing data orenhancements to the various alternative OTT CAS system embodiments andthe alternative methods of using the same. In the exemplaryrepresentations of FIGS. 60-62B, a number of different sensor locationsare provided. More or different locations are possible as well as theplacement of sensors in each of the illustrative locations in differentorientations or having multiple types of sensors or of the same type ofsensor in one location.

Moreover, for each embodiment of a sensor enabled OTT device, eachsensor location utilized has a corresponding modification to the housing110/210, electronics 130, 230 along with the related specifications anddetails of FIGS. 5-15B as needed based on the number and type, ornumbers and types of sensors employed in that embodiment. In addition,the OTT device is also modified and configured to provide as needed theappropriate number and type of electronic mounts, mechanical orstructural supports, electronic or vibration insulation, electrical/dataconnections, hardware, software, firmware and all related configurationsto provide for operation and utilization of each sensor type. The type,number and location of sensors on an OTT device are employed in order toprovide enhanced information about the OTT device and/or CAS operatingenvironment in conjunction with other tracking and operating parametersalready employed by the OTT CAS system and described herein.

In various alternative operating schemes of utilizing a sensor enhancedOTT device, the OTT CAS system operations, decision making, modeselection and execution of instructions is adapted based upon theaddition of data from one or more OTT device sensors to provide one ormore of: position, movement, vibration, orientation, acceleration, roll,pitch, and/or yaw, each alone or in any combination as related to theOTT device itself or the surgical tool under OTT tracking and guidance.Still further, multiple sensors or detection or measurement devices ofthe same type may be placed on the OTT device in different positions andthen those same input types from each of the different locations mayalso be used to provide additional OTT CAS operational inputs,determinations or control factors. Each of the separate sensor outputsor readings may be used individually or the data from the same types ofsensors may be collected together and averaged according to the type ofsensor and data use. Still further, the collection and use of sensordata (i.e., sampling rate, weighting factors, or other variables appliedbased upon hover mode state, and/or adjustment of one or more CAS systemparameter) may be adjusted according to the various operational schemesdescribed in FIGS. 31A-36 and in particular with regard to adjustmentsto operating parameters such as slew rate and data collection rates asdescribed in FIG. 63.

Turning now to FIG. 60, there is shown a top view of an embodiment ofthe OTT device 200 with the top of housing 205 removed. Sensor locations1, 2, 3, 4, 5 and 6 are seen in this view. Sensor locations 1 and 2 areoutboard on either side of the OTT device centerline. In thisembodiment, the sensor locations 1, 2 are adjacent to the cameras 215.An additional sensor location 3 is illustrated in the central portion ofthe OTT device. The sensor location 3 may be positioned in, for example,the geometric center of the OTT device, at the center of mass or gravityof the OTT device, or at the center of mass or gravity for the combinedOTT device/tool. The location of sensor position 3 may therefore bechanged based on the type of tool 50 attached to the OTT device. Inaddition or alternatively, for OTT device embodiments configured tooperate with a variety of different tool types, a corresponding numberof appropriately positioned sensors may be placed depending upon thespecific type of tool used. In these embodiments, the OTT CAS system isalso configured to recognize or receive input as to the type of toolattached to the OTT device and then select or utilize the output fromthe sensor or sensors in the sensor locations and sensor typesassociated with that particular tool configuration.

Sensor locations 4 and 5 are positioned towards the rear on the left andright outer edges of the OTT housing 205. Sensor position 6 is on thecentral portion near the rear of the housing 205. The use of sensorlocations 1, 2, 4, 5 and 6 alone or in any combination may be used inobtaining one or more or roll, pitch, or yaw angle data as well andinclination and/or multiple axis movement rates or vibration reading ineach of these locations.

FIG. 61 is a perspective view of the OTT housing 205 of the view of FIG.60. From this view, the sensor location 3 can be seen in its point nearthe center of the system. Sensor position 7 that is internal to thehousing 205 is shown in phantom along the housing left side. The sensorposition 7 is on or within the left wall portion towards the rear of theOTT housing 205. FIG. 61, illustrates the coordinate position of sensorlocation 7. In this illustrative example, the sensor location 7 is shownrelative to a central OTT location, here sensor location 3. Anyreference point may be used by the OTT CAS system directly or through asensor driver intgermediary computer for coordination and crossreference of the various sensor inputs. In this example, the sensorlocation 7 is—relative to the central location 3—spaced rearward by adistance of d. In addition, the sensor location number 7 is spaced by aheight h from the elevation of the sensor location 3. The specificlocation of each one of the sensors may be used to advantage whendetermining the various parameters of the OTT in use. It is to beappreciated that the OTT CAS system may use absolute x, y, zcoordinates, or relative coordinates for the sensor locations employedby an OTT device embodiment.

FIG. 62A is a similar isometric view to that of FIG. 61 with the lowerOTT housing portion removed. The view of FIG. 62A is used to illustrateseveral additional optional sensor locations. Sensor locations 8, 9, 10,11 and 12 are shown in this embodiment. Sensor locations 12, 9 and 8 areshown along the central longitudinal axis of the OTT device fore and aftof the central sensor location 3. Sensor locations 10, 11 provideadditional outboard locations similar to positions 4 and 5 butlongitudinally separated therefrom. While many of these exemplarylocations are shown along or near the longitudinal center line of theOTT device, other sensor locations are possible. For example, sensorsmay also be located on the underside of the board 235 or other structurewithin, part of or attached to the OTT device housing. The sensorlocations may be placed in, along, above or below the board 235 or inother locations based on design and space requirements for othercomponents and the OTT device electronics package.

In addition to the sensor locations described in FIGS. 60, 61, and 62A,a sensor platform 20 may also be provided within OTT housing 205. Aperspective view of an exemplary sensor base 20 is illustrated in FIG.62B. The sensor base 20 is shown with representative sensor locations 1,2, 13, 14, 15, 16, 17, 18 and 7. The sensor base 20 illustrates thealternative placement of sensor 7 on the base 20 instead of within or onthe wall in FIG. 61. Similarly, sensor positions 1 and 2 are moved fromthe positions illustrated in FIG. 60 to the base 20. In addition, thelocation of sensor position 15 is selected to provide the functions ofsensor location 3 described above. The various alternative sensor types,numbers and locations may be integrated as described above into anappropriately configured sensor base 20. In various implementations, onesensor base or more than one sensor base may be sized as shown in FIG.62B where the sensor base mimics the size and shape of the OTT devicehousing 205. A sensor base may include all the sensors of a particulartype, particular orientation, for a particular location or position orfunction related to the particular OTT device configuration. Given therate of miniaturization of electronics and sensors, particularly in thefield of micro electrical mechanical systems (MEMS), it is to beappreciated that all or substantially all of the sensors employed in anOTT device may be in the form of suitably miniaturized commerciallyavailable components.

FIG. 62B shows the sensor locations 13 and 14 corresponding to cameralocations and forward of sensor locations 1, 2. Sensor positions 13, 14,1 and 2 are provided in proximity to the camera locations. Sensorlocations 15, 16 and 18 are near the center line of the OTT devicemodule when the sensor board 20 is in place. Sensor locations 15 or 16may be positioned above a specific location of interest in the OTTguided tool such as a vertical central axis of the tool, triggerlocation or other feature of interest to facilitate tracking of thattool. In one aspect, a sensor location is positioned to indicate thetrigger of the surgical tool being used in the CAS system. In oneembodiment, sensor locations 17 and 7 are positioned to the left andright outboard positions behind the center of mass for the tool. Sensorlocation 18 is the rearward sensor location furthest to the rear of theOTT module when the sensor board 20 is installed into the OTT housing205.

Each one of the sensor locations illustrated and described withreference to FIGS. 60-62B and elsewhere in this specification, may beused to provide a variety of different sensor or instrumentation typesto be used by the position and tracking systems described herein. By wayof example and not limitation, the various instruments or sensors usedin conjunction with an OTT device include: an inclinometer, a gyroscope,a two axis gyroscope, a three axis gyroscope or other multiple axisgyroscope, an one-two-three or multiple axis accelerometer, apotentiometer, a MEMS sensor or micro-sensor or MEMS instrumentconfigured to provide one or more of roll, pitch, yaw, orientation, orvibration information related to the OTT device, or the operation of anOTT device/surgical tool combination or the operation, use or status ofa tool attached to an OTT device and being used under an OTT CAS systemas provided herein or as otherwise used in an operating environment ofthe OTT system for tool or prosthetic registration, fit assessment orsurgical planning, surgical plan revision and the like.

FIGS. 16A, 16B and 16C provide various views of a reference frame 300for use in a computer assisted surgery procedure. There is a 305 framehaving a planar or general 3D surface 310 bounded by perimeter 315. Oneor more active or passive fiducial marker 70 are arranged in a pattern72 across the surface 310 or carried individually through some framestructure. There is a stem 320 extending from the frame 305 and acoupling 325 on the stem. The coupling 325 is used to join the frame 305to a base 330. The base 330 has a first surface 335 configured to engagea portion of the anatomy within a surgical field related to theprocedure. The base 330 has a second surface 340 to engage with thecoupling 325. The coupling 325 and the second surface 340 are engaged inFIG. 16A but are separated in FIGS. 16B and 16C. In the views of FIGS.16C and 16C at least one registration element is visible on the couplingand at least one registration element is visible on the second surface.In the illustrated embodiment, the registration element 342 b is afemale feature on the coupling 325 while the coupling element 325 a onthe second surface 340 is a male feature. The registration elements aresized and positioned to mating cooperation when the coupling 325 and thesecond surface 340 are engaged. It is to be appreciated that a varietyof different registration element types and positions may be adapted andconfigured for providing mating cooperation when the coupling is engagedto the second surface.

The base 330 includes a second surface 335 used to engage the anatomy.All or a portion of the surface may include a serrated edge to assist inengaging with anatomy, particularly bony anatomy about the joint. Thebase first surface 335 comprises a curvature that is complementary tothe anatomical site upon which the base first surface is to be affixedduring the surgical procedure. In one aspect, the curvature iscomplementary to an anatomical site comprising a skin portion of theanatomy, where the bone may not be exposed but the reference frame isattached to it through the skin with screws or other fastening devicementioned below. In one additional embodiment, the bony portion of theanatomy is adjacent to a joint that is the subject of the surgicalprocedure. The joint may be selected from a knee, a shoulder, a wrist,an ankle, a hip, a vertebrae or any other surgical site where a boneosteotomy is to be performed. The base 330 includes at least oneaperture 337 adapted and configured for a fixation element used to affixthe base to a site on the body. The fixation element may be selectedfrom one or more of a pin, a screw, a nail, surgical staple or any formof glue or cement to be applied to the element or to be exposed (e.g.,peeling of a double sided tape).

FIG. 17 illustrates an isometric view of the reference frame guide 350.The reference frame guide 350 has a frame 355 and a stem 360 extendingfrom the frame 355. The stem 360 has a curvature or shape configured toengage with an anatomical feature to assist, when the frame guide isattached to the frame 305, the reference frame 300 is placed in adesired position and orientation within the surgical field. Thereference frame guide 350 also includes one or more engagement elements365 along the frame 355 for temporary engagement with the perimeter 315or a portion of the reference frame 305 to permit proper positioning andadjustment of a base 330 associated with a reference frame 300 attachedusing the elements 365. FIG. 18 illustrates a reference frame guideattached to the frame 305 of a reference frame 300. In use, theengagement elements 365 may be broken off in order to remove thereference frame from the guide frame during surgical procedure. Whileillustrated in mating cooperation with reference frame 300, referenceframe guide 350 may be adapted and configured to form a matingengagement with reference frames of different shapes and sizes, such asthe reference frame 400 in FIG. 24.

In one particular embodiment, the curvature or shape 362 of the stem 360is configured for placement of the stem in relation to the condyles inorder to provide alignment within the surgical field for the referenceframe 300 along the femur. Positioning of the base 330 along the femur10 is shown in FIGS. 19 and 20. The joint reference frame guide andreference frame structure (see FIG. 18) are positioned (following thearrow in FIG. 19) so as to align the curvature 362 of the stem 360between the condyles 12 of the femur 10 in order to place the base 330in proper orientation on the femur as shown in FIG. 20. Thereafter thereference frame 300 is attached to the femur 10 by joining the basefirst surface 335 using one or more methods such as and screws or nailsapplied the aperture 337 or the use of a biocompatible bone cement. Oncethe reference frame 300 is confirmed in the proper position, thereference frame guide 350 is removed (FIG. 21) leaving only thereference frame in the desired location along the femur 10 in thedesired relation to the condyles 12 according to a surgical plan to beimplemented (FIG. 22).

FIG. 23 illustrates an embodiment of the reference frame 400 andposition along the tibia 15. In this illustrated embodiment thereference frame 400 is attached on or about the tibial tuberosity (shownmore clearly in FIG. 25) and secured to the bone using any one of theseveral fixing methods described above with regard to the referenceframe 300. Additional details of the reference frame 400 may be providedupon review of FIGS. 24A, 24B and 24C. These figures provide variousviews of a reference frame 400 for use in a computer assisted surgeryprocedure. There is a 405 frame having a surface 410 bounded byperimeter 415. One or more active or passive fiducial markers 70 arearranged in a pattern 74 across the surface 410. There is a stem 420extending from the frame 405 and a coupling 425 on the stem. Thecoupling 425 is used to join the frame 405 to a base 430. The base 430has a first surface 435 configured to engage a portion of the anatomywithin a surgical field related to the procedure. The base 430 has asecond surface 440 to engage with the coupling 425. The coupling 425 andthe second surface 440 are engaged in FIG. 24A but are separated inFIGS. 24B and 24C. In the views of FIGS. 24C and 24C at least oneregistration element is visible on the coupling and at least oneregistration element is visible on the second surface. In theillustrated embodiment, the registration element 442 b is a femalefeature on the coupling 425 while the coupling element 425 a on thesecond surface 440 is a male feature. The registration elements aresized and positioned to mating cooperation when the coupling 425 and thesecond surface 440 are engages. It is to be appreciated that a varietyof different registration element types and positions may be adapted andconfigured for providing mating cooperation when the coupling is engagedto the second surface.

The base 430 includes a second surface 435 used to engage the anatomy.All or a portion of the surface may include a serrated edge to assist inengaging with anatomy, particularly bony anatomy about the joint. Thebase first surface 435 comprises a curvature that is complementary tothe anatomical site upon which the base first surface is to be affixedduring the surgical procedure. In one embodiment, the bony portion ofthe anatomy is adjacent to a joint that is the subject of the surgicalprocedure. The joint may be selected from a knee, a shoulder, a wrist,an ankle, a hip, or a vertebrae. The base 430 includes at least oneaperture 437 adapted and configured for a fixation element used to affixthe base to a site on the body. The fixation element may be selectedfrom one or more of a pin, a screw, a nail, a surgical staple or a glueor adhesive based fixation.

Turning now to FIGS. 26A, 26B and 26C, additional aspects of thereference frame designed to be described. With reference to FIG. 26A,the orientation between the frame 305 and the base 300 may be adjustedbetween a number of preset orientations. Altering the relationshipbetween these two components is accomplished by altering which of aplurality of registration elements available to the joint as componentsare engaged. In one aspect, there are a plurality of registrationelements on the coupling and a plurality of registration elements on thesecond surface. The orientation of the reference frame may be adjustedbetween a first orientation 382 and a second different orientation 384based on which grouping of registration elements is used for joining thebase 330 to the frame 305. In one embodiment, wherein a portion of theregistration elements on the coupling are engaged with a portion of theregistration elements on the second surface the result will orient theframe in a first orientation within the surgical field. In anotheraspect, the mating different registration elements on the coupling withdifferent registration elements on the second surface, the result isthat the frame 305 will present in a second, different orientationwithin the surgical field. In one aspect, the first orientation is aknown position used in surgical preplanning. In still another aspect,the second orientation is another known position used in surgicalpreplanning. Either or both of the first orientation and the secondorientation may be used in furtherance of the OTT CAS techniquesdescribed herein. Both can be used in sequence without new softwareregistration each time. The registration for each configuration or onlyone is done first and once, and the software registration for the otheris computed from the geometry or measured separately and its data storedand accessible whenever needed.

FIG. 26A also illustrates one embodiment of a mount coupling adapted andconfigured to maintain the relative position and orientation of thecoupling and the second surface. In this embodiment a flexible linkage380 is shown between the two components and is sized shaped and orientedwithin the reference frame to maintain the orientation of the frame 305within the surgical field. In other words, the mount coupling issufficiently rigid that if the frame 305 is bumped during a procedure,its components can be temporarily displaced relative to each otherthrough deformation of the elastic element in the coupling, but then canreturn back or be returned back by the user to the original alignment,and so it will not lose its alignment due to the registration elementswithin it. If the bump of the reference frame was sufficiently strong,the registration lements would disengage and not return automatically,but the use can return them and the original software registeredalignment is still not lost. In the illustrative embodiment, theflexible linkage 380 is disposed completely within the structure in use,here the base 330. As best seen in FIG. 26A, one portion of the linkage380 attaches to the upper base 330 and another portion to the lower base330. In another alternative aspect, a mount coupling is provided in sothat when the mount coupling is attached to the reference frame themount coupling substantially or completely surrounds the area of matingcontact between the coupling and the second surface. FIG. 26B1 aillustrates a perspective view of a flexible mount coupling 383 thatcompletely surrounds the interface between the upper and lower base 330.FIG. 26B1 b illustrates a perspective view of the flexible mountcoupling 383. FIG. 26B2 a illustrates a perspective view of a flexiblemount coupling 384 that substantially surrounds the interface betweenthe upper and lower base 330. The coupling 384 includes four cornermounts connected by linkages. The corner mounts and linkages are—likecoupling 383—designed for a snug fit around the interface between theupper and lower mounts. FIG. 26B2 b illustrates a perspective view ofthe flexible mount coupling 383.

FIGS. 27A and 27B provide alternative reference frame surface shapes aswell as alternative height to show marker patterns. FIG. 27A illustratesa generally rectangular frame 390 of a reference frame having aplurality of fiducial markers 70 arranged in a pattern 78. FIG. 27Billustrates a generally trapezoidal surface shape 310 on the frame 395.A plurality of fiducial markers 70 arranged in a pattern on the surface305.

FIG. 28 illustrates an isometric view of a representative of prosthesis20 for use in a total knee replacement procedure. The numbers indicatedon the prosthesis 20 are representative of the types of cuts undertakenduring knee surgery. FIGS. 29A-29I and 30 illustrate one of the uniquecombinations of the OTT CAS system described herein. While each of thereference frames described above may be used independently or inconjunction with other anatomical sites or surgical equipment, thereference frames 300 and 400 have particular advantage for the on tooltracking devices and OTT CAS procedures described herein. One challengeof using on tool tracking devices for handheld precut surgery isobtaining relevant tracking information and maintaining a tracking frameof reference during the procedure. By the unique design and placementthe reference frames 300 and 400 may be used to provide just this typeof dynamic reference frame tracking using the OTT tracking techniquesdescribed herein. As shown in the figures that follow in each one of therepresentative cuts used for implanting the prosthetic 20, the visionsystem carried onboard the OTT 100 is able to visually identify andregister with all or a portion of the reference frame 300 and thereference frame 400. While these particular configurations areillustrative of the capabilities of the OTT CAS system and tools forknee surgery, it is to be appreciated that the reference frames andvision guidance techniques described herein may be adapted to otherjoints in the body and to other procedures.

FIGS. 29A-29I and 30 each illustrate a representative surgical set upfor the placement of a reference frame 300 on the femur 10 and thereference frame 400 along the tibia 15, in particular on or about thetibial tuberosity 18. Is to be appreciated that the illustrated OTT CASprocedure that follows utilizes the reference frames 300, 400—they arenot moved but remain in the same position during all of the followingOTT CAS process steps. An on tool tracking device 100 is coupled to asurgical tool 50 for the positioning and use of a tool 54 having anactive element 56.

In the illustrative embodiment of FIG. 29A, the OTT 100 is providingguidance for the use an active element 56 for making a distal lateralcondyle cut. During this cut, the cameras carried onboard OTT 100 arecapturing, imaging, and providing relative navigation and positioninginformation based on information received from both reference frames 300and 400 during all or a substantial portion of the illustrated cut.

In the illustrative embodiment of FIG. 29B, the OTT 100 is providingguidance for the use an active element 56 for making a distal medialcondyle cut. During this cut, the cameras carried onboard OTT 100 arecapturing, imaging, and providing relative navigation and positioninginformation based on information received from both reference frames 300and 400 during all or a substantial portion of the illustrated cut.

In the illustrative embodiment of FIG. 29C, the OTT 100 is providingguidance for the use an active element 56 for making an anterior cut.During this cut, the cameras carried onboard OTT 100 are capturing,imaging, and providing relative navigation and positioning informationbased on information received from both reference frames 300 and 400during all or a substantial portion of the illustrated cut.

In the illustrative embodiment of FIG. 29D, the OTT 100 is providingguidance for the use an active element 56 for making a posterior lateralcondyle cut. During this cut, the cameras carried onboard OTT 100 arecapturing, imaging, and providing relative navigation and positioninginformation based on information received from both reference frames 300and 400 during all or a substantial portion of the illustrated cut.

In the illustrative embodiment of FIG. 29E, the OTT 100 is providingguidance for the use an active element 56 for making a posterior medialcondyle cut. During this cut, the cameras carried onboard OTT 100 arecapturing, imaging, and providing relative navigation and positioninginformation based on information received from both reference frames 300and 400 during all or a substantial portion of the illustrated cut.

In the illustrative embodiment of FIG. 29F, the OTT 100 is providingguidance for the use an active element 56 for making an anterior chamfercut. During this cut, the cameras carried onboard OTT 100 are capturing,imaging, and providing relative navigation and positioning informationbased on information received from both reference frames 300 and 400during all or a substantial portion of the illustrated cut.

In the illustrative embodiment of FIG. 29G, the OTT 100 is providingguidance for the use an active element 56 making a posterior lateralcondyle chamfer cut. During this cut, the cameras carried onboard OTT100 are capturing, imaging, and providing relative navigation andpositioning information based on information received from bothreference frames 300 and 400 during all or a substantial portion of theillustrated cut.

In the illustrative embodiment of FIG. 29H, the OTT 100 is providingguidance for the use an active element 56 making a posterior medialcondyle chamfer cut. During this cut, the cameras carried onboard OTT100 are capturing, imaging, and providing relative navigation andpositioning information based on information received from bothreference frames 300 and 400 during all or a substantial portion of theillustrated cut.

In the illustrative embodiment of FIG. 29I, the OTT 100 is providingguidance for the use an active element 56 making a tibial cut. Duringthis cut, the cameras carried onboard OTT 100 are capturing, imaging,and providing relative navigation and positioning information based oninformation received from both reference frames 300 and 400 during allor a substantial portion of the illustrated cut.

FIG. 30 illustrates an OTT 100 coupled to a surgical instrument 50having a tool 54 and an active element 56. Reference frames 300, 400 arealso shown in relation to a OTT CAS surgical site about the knee. Anadditional reference frame 397 having a stem 398 and tip 399 is beingused for further registration or notation of the surgical field. Theregistration of the reference frame 397 is being provided by the imagingsystem of the OTT 100 mwith a tool. The registration frame 397 is beingregistered along with one or both of the registration frames 300, 400.While embodiments of the OTT CAS methods described herein by utilizeboth the reference frames 300, 400, it is to be appreciated that the,because of the improved image based tracking capabilities of the OTT andOTT CAS processing the OTT CAS system have both reference framesavailable but elect during processing to only use tracking informationfrom one reference frame.

When considering the use of the unique reference frame embodimentsdescribed herein, consider the manner by which a view may be preferredby an OTT CAS system user. The OTT CAS system is pre-programmed so thatcertain views are shown by default for certain cuts. For instance, inthe example of resecting a femur in preparation for a femoral prostheticfor a TKR procedure, several surfaces are to be cut, as shown in FIGS.29 and 30. Each surface may be best viewed from a different perspectiveduring the procedure. When cutting the anterior surface of the medialcondyle a first view may be desirable, whereas when cutting the anteriorsurface of the lateral condyle a second view may be desirable.Accordingly, the system sets a pre-defined first view for viewing thevirtual model when the anterior surface of a medial condyle is resected.Similarly, default visual views can be defined for a number of commonresection procedures. When the OTT CAS system determines the cut to beperformed, the system determines the best match for the cut and displaysthe default automatically without the intervention of the surgeon. Inmuch the same way the vision based processes performed by the OTT CAScomputer may be preselected to use all or a portion of the availabletracking information from one or both reference frames, automatically,depending upon the circumstances. In addition, the OTT CAS may guide auser in adjusting orientation of a reference frame within a surgicalfield to improve guidance information from that frame. The adjustableorientation of the frame while maintaining the registration position ofthe base is described herein.

In another alternative aspect, there is a divot or other feature presenton one or more of the reference frames described with reference to FIGS.16A-30. In one aspect, contact is made with the divot using the surgicaltool, touch screen, or navigated pointer and produces a result in thesystem indicating the initiation or completion of a step. In oneexample, contact with the reference. frame (e.g., touching with anavigated pointer) the OTT CAS system registers the initiation of anoperation or alternatively the completion of an operation. In onespecific embodiment, the act of touching the reference frame indicatesthe start of an operation involving that particular reference frame. Oneexemplary operation conducted with a reference frame is boneregistration. In an additional aspect, this input and/or interactionwith a particular reference frame is also an input to or part of aselection criteria for a CAS Hover mode, smart view, display or otherfunction.

It is to be appreciated that any of a number and variety of powered ornon-powered tools can be utilized with the OTT CAS systems describedherein. For example, in the orthopedic surgery field, the system can bebuilt upon a single orthopedic power saw such as a Stryker System 6Precision Oscillating saw. Similarly the system can be used with otherpower tools commonly used in orthopedic surgery, such as a burr or adrill. In such application, the system could be integrated within thedesign of the surgical tool, or added as a retrofit. In addition, thesystem could utilize a tool that does not require any external powersource—such as a pointer, a marker or a scalpel. Ideally, the systemcould accommodate multiple smart tools to be used at different phases ofa surgical procedure and make the system robust enough to perform a widevariety of surgical procedures. It is to be appreciated that the OTT 100may be adapted to fit the housing of a wide variety of surgical tools,free hand tools as discussed above and elsewhere in this application.Alternatively, the OTT may be built (fully integrated) into the designof freehand tools or hand-held power instruments and its housingmanufactured together with such tools. Additional OTT housingconfigurations such as various two part housings are illustrated anddescribed below with reference to FIGS. 68 a-72.

The system could be used in other applications outside of orthopedicsurgery. For example, it could be used in simulations and simulators forteaching and training surgeons for orthopedic surgery. Alternatively thesystem could be used for other medical procedures that require preciseorientation and manipulation of rigid tissue. The present techniquescomputer assisted surgery could readily facilitate such dentalprocedures. The system can also be used in non-medical applications, forexample in carpentry, sheet metal work and all other engineering markingand machining processes to guide the user to make a certain pattern ofcutting or drilling of materials.

Embodiments of the OTT CAS system described herein eliminates the needfor external tracking devices by placing one or more trackers on boardthe tool. The present invention can completely eliminate the need for anexternal tracking system or utilize the tracking sub-system to add newtracking data. In either configuration, the tool itself tracks thepatient's anatomy, or tracks itself relative to a patient anatomy, asopposed to an external tracker that tracks both to determine therelative position of one to the other. Furthermore, because thecomponents providing input to the tracking system are located on thetool itself, all tracked elements of the system are tracked relative tothe tool. As a result, the tracking data produced by the on-tooltrackers is very different. The position of the tool, for example, neednot be independently tracked because all other tracked objects aretracked from the tool's vantage. The on board tracking system alleviatesconcerns faced by externally tracked systems, where all components ofthe system including the surgical instrument are tracked by an externaldevice. Logistically, the present invention allows the operating room toeliminate or at least minimize the need for a separate piece ofequipment in the operating room by placing the tracking or thecomponents providing input to the processing part of the tracking systemon the tool itself. With the sensors for the tracking on board the tool,this brings another advantage of being closer to the tracked target, andthus higher resolution and accuracy may result as well as less stringentrequirements for “line of sight” access between the tracker and thetracked element of other systems.

The tracker-tracking subsystem further comprises one or more trackingelements that are detectable to the trackers on board the surgicalinstrument. There are a wide variety of tracking elements that can beutilized in the system. For example, reference frames that contain oneor more reflective surfaces can reflect infrared or visible light backto the surgical tool. Light emitting diodes can similarly indicate theposition of tracked objects back to the surgical tool. Other approaches,such as fiducial points or image recognition, could eliminate the needfor external reference frames to be placed on the objects, such as thepatient's tissue, that needs to be tracked. In further embodiments, thespecific image of the patient's anatomy can serve as the trackingelement without the aid of any other reference points.

The surgical instrument tracks the position of the tracked element bymeans of one or more trackers. In one embodiment, the system utilizesstereoscopic placement of two cameras as the tracker. The cameras areside by side, tilted at a range of angles suitable for stereo-vision, oneither side of the saw's blade/drill-bit/burr, etc. For other tools,such as a drill, the cameras can similarly be placed stereoscopically,side by side, on either side of the drill bit or any other tool's endeffector.

The placement of the cameras, relative to the end effector of the tool,impacts the operation of the tracker-tracking element subsystem. Forexample, placement of the camera or cameras far back from the endeffector expands the field of view. For applications like jointreplacement, or when the tool is in close proximity to the patient'sanatomy, a wide field of view is helpful. With an expanded field ofview, the tool can find the tracking element more easily. Placing thecamera or cameras closer to the tool's end effector constricts the fieldof view, but adds magnification and resolution useful for applicationssuch as dental surgery. In addition, placement of the camera must takeinto account the relative position of the other elements of thesubsystem. Placing the cameras so their axes are in the plane of the endeffector of the tool would minimize the extent to which the end effectorblocks the view of the cameras. It is contemplated, however, that thecameras may be placed in any configuration that is deemed appropriatefor tracking one or more tracking elements in a surgical procedure. Astechnology advances, configurations beyond those currently described maybe more favorable in regards to particular tools and surgicalenvironments.

The sub system can utilize a wide variety of cameras or systems ofcameras. Generally, the system utilizes digital cameras. In addition,the system utilizes at least two cameras to provide stereoscopic vision.It is possible to use analog cameras, provided there was effective meansof digital conversion such as the established technology of image formatconversion which are sometimes known as ‘frame grabbers’ or ‘capturecards’. Stereoscopic vision, and the ability to gain further informationbased on the differences in the images from the two cameras, helps thesystem to better locate the tracking element in three dimensions interms of position and orientation or pose. Systems could utilize morethan two cameras utilizing what is known as “redundancy” to improve theability to navigate, such as in the cases when some of the trackedelements are not visible to one or more of the cameras and thus twocameras would not suffice in those instances. Additionally, a systemcould utilize a single camera but would need additional image processingto navigate as accurately as a stereoscopic system.

Alternatively, the subsystem could utilize a different system oftrackers and tracking elements. In one alternative, the tracker is ahigh-resolution camera optimized for image recognition under the visiblelight spectrum present in standard Operating Room conditions. Thetracking element is the patient's anatomy, based on the medical imagestored in the surgical plan. In addition, a narrower field of view mayalso benefit the efficient recognition of the patient's anatomy.Finally, the surgical plan itself may need to incorporate or identifyparticular anatomical landmarks of the patient to establish functionaltracking elements.

Regardless of configuration, the cameras need to have sufficientresolution to accurately track the tracking element to a certainpredetermined level of accuracy. For example, a system with a trackingelement that is a reference frame with infrared LED's, cameras with640×480 resolution have sufficient resolution to track the trackingelement with surgical accuracy. Systems can utilize additional elements,such as infrared filters, and isolate the tracking element for thecameras. A lower resolution camera, in such a system, can be sufficientto produce highly accurate tracking.

Resolution is not the only characteristic of the cameras that influencesthe operation of the system. The frame rate is an importantconsideration, depending upon the particular configuration of thesystem. For example, a very high frame rate of around 100 Hz (frames persecond) would produce minimal latency but would be very burdensome onthe image processor. The system would require a powerful processor inorder to extract the tracking element from so many captured images in agiven unit of time. Alternatively, if frame rate is too low then thesystem will produce too much latency. If the operator were to move thetool too quickly then the system would not be able to continuously trackthe tool. The minimally acceptable frame rate should be utilized in thesystem. For a system that utilizes infrared LED's in the reference framealong with an array of VGA cameras, a frame rate of 30 Hz would producea system suited to freehand orthopedic surgery.

Together, these examples illustrate a variety of configurations for thetracking element and the cameras that comprise the exemplarycamera-tracking embodiments of the tracker-tracking element subsystem.In addition to the accurate placement of the tracking element, thetracking element's location must be extracted from the images capturedby the camera. An image signal received from the cameras must undergodigital signal processing (DSP) to convert the image of the trackingelement to mathematical coordinates, relative to the tool. Themathematical coordinates are then sent to a computer system and comparedagainst the surgical plan, allowing the computer system to determine ifthe surgical path is following the intended resection.

Consider that there are several steps to process the raw data from thecameras into the mathematical coordinates. Initially, the system mustacquire the image. For the camera detecting the markers (e.g. infraredLED's, reflecting bodies, fiducials, etc.), the system must: determinethe coordinates of the centroid of each of each individual marker usedin the overall tracking element, determine the sizes of each element,and report the size and shape and the coordinates of each LED to thecomputer system. Additional operations to process the captured image,such as sub-pixel analysis to determine the location of the centroid canimprove accuracy.

For systems that operate at 30 Hz, steps must be completed inapproximately 33 ms, and the computer will need to determine therelationship between the individual LED's and calculate the position andorientation of the tracking element. From that data, the computer willhave to determine the orientation of the model and the relativepositions between the bone and the surgical tool. The signal processingonly has the amount of time between two successive frames to perform anyneeded operations. (For example, for a frame rate of 30 Hz, theprocessing system has the above mentioned 33 ms period to perform theseoperations) In one embodiment, the majority of the forgoing steps can beaccomplished on the tool itself often by integrated CPU's on the cameras(or other trackers) themselves.

For example, additional processing of images captured by the cameras canbe accomplished via a CPU that is integrated into the camera, or on thecomputer system or some combination of the two. For example, many smallcameras have integrated CPU's capable of running digital signalprocessing algorithms prior to exporting the data signal. The DSP cancomprise a simple step, like converting color images to grayscale orcomplex operations, like cropping the video image to a small box thatsurrounds the identified LED's. The initial processing makes the finalextraction of the tracking element from the images captured on thecamera less computationally burdensome and the overall tracking processmore efficient.

The camera-tracking element subsystem can either utilize digital cameraswith digital image transmission, or with wireless transmission. There isa wide variety of cameras with digital image transmission which aregenerally termed “IP” or “Wifi” cameras. Many small, low cost solutionscan be used, streaming images (which can be synchronized between twocameras) in any format (e.g. Mpeg) and fed to the processing electronicsthrough one of many known digital streaming protocols. Alternatively,analogue Image transmission can used as has been in model airplanes withwhat is known as First Person View (FPV) technology. This facilitatesreadily available commodity cameras, with minimal weight and size, smallwireless transmission and low cost. After image processing andextraction of the coordinates for the tracked elements, additionalprocessing is necessary to create tracking data sufficient to inform thecomputer system. The coordinates of the tracked elements are combinedwith information about the cameras (such as the specifications andcalibration data) to further refine the location space of each trackedelement. Based on the refined location of each tracked element, the subsystem utilizes user-defined definition of clusters for the particulartracking element (sometimes called a reference frame) to detect validclusters for the tracking element and their position and orientation inspace. The data determining position and orientation in space is theformatted for use. For example, the system can place the specialcoordinates into a matrix that is compatible with the overall definitionof the space used in a surgical plan.

The forgoing processing is different from the processing that can occuron the tool and is not image conditioning and spatial extraction. It canbe processed through dedicated software that could be in the samecomputer system where the surgical plan and planned resection iscomputed or it could happen on an intermediary computer that could be onthe tool or separate from both the tool and the computer system.

Additional navigation data can augment the camera-tracking elementsystem. The tool can further contain one or more accelerometers orinertia sensors to determine the orientation and movement of the toolalong the surgical path. The accelerometers can provide additional datato the computer system, in addition to the tracking data from the cameraor cameras. Alternatively, an external tracking system can augment theon-board tracking of the tool. No such application is required but canserve to augment the tracking capability of the system mainly by‘anticipating’ the movement of the user. Systems could further includemultiple tracker-tracking element modalities. For example, the systemcould include an infrared camera and a tracking element with an infraredLED as well as a visible light camera for optical resolution. Trackinginformation from both could be processed to establish the coordinates ofthe tool in three dimensions.

As is typical in computer aided surgery, a surgical plan is determinedbefore commencing the desired surgical procedure or prior to performinga step in the desired surgical procedure. The surgical plan is based onintended resections designated by the surgeon on a computer rendition ofa patient's anatomy. A computer rendition of a patient's anatomy may beprocured through a variety of medical imaging techniques, such as CT orMRI scanning. In addition, a computer rendition of a saw, drill, burr,implant, or any surgical instrument or part thereof may be procured bydesign specifications (or models) programmed into the computer system.Once a computer rendition of patient's anatomy is accessible through acomputer interface such as a display, mouse, keyboard, touch display, orany other device for interfacing with a computer system, the surgeon maymanually designate resections for the surgical plan by entering one ormore cuts to be performed, a region to be drilled, or a volume of tissueto be removed into the computer system. Alternatively the computersystem may be configured to generate the surgical plan based on a set ofspecified parameters selected by the surgeon. The specified parametersmay correspond, for instance, to the shape, size, and/or location of animplant that the surgeon wishes to attach to the patient's anatomy. Thecomputer may accordingly generate a surgical plan comprising theresections necessary to fit the implant to the patient's anatomy. Oncethe surgical plan is designated by the surgeon, the computer systemtranslates the surgical plan into one or more mathematically definedsurfaces defining the boundaries of the intended resections thatcomprise the surgical plan. Data acquired by the previously describedtracker-tracking element subsystem can then be used to compare theinstrument's surgical path with the surgical plan in order to determinethe deviation of the surgical path.

Next, the surgical plan is delineated as one or more surfacesmathematically defined in an acceptable three dimensional coordinatesystem such as Cartesian, spherical, or cylindrical coordinates, orother anatomically based coordinate systems. For example, in a surgicalplan that uses Cartesian coordinates, a cut may be defined as aspecified distance along each of the X, Y, and Z axes from an XYZcoordinate defining the origin. The specified distances along each axisneed not be linear. For example, a cylinder representing a region to bedrilled in the patient's anatomy may be defined in Cartesian coordinatesas a circular surface having a specified diameter located around anorigin and protruding for a specified distance from the origin in adirection that is perpendicular to the circular surface. Any cut, seriesof cuts, or volume of tissue to be removed may be mathematically definedthrough a similar approach of defining surfaces that delineate theboundaries of the surgical plan that the surgical instrument must followto complete the designated resections.

As previously noted, the surgeon may manually designate the resectionsof the surgical plan on a computer rendition of the patient's anatomy.In one embodiment the surgeon can use the computer interface to view andmanipulate a three dimensional rendition of the patient's anatomy andmake marks representing cuts. The marks made on the three dimensionalrendition are then translated into the mathematical surfaces delineatingthe surgical plan that the surgeon must follow with the surgicalinstrument.

In surgical procedures utilizing implants such as a total kneereplacement surgery, it is advantageous to use the physicalspecifications of the implant when delineating the surgical plan forbetter assurance that the implant will fit onto the patient's anatomycorrectly. In such an embodiment, the surgeon can use the computerinterface to view and manipulate a three dimensional rendition of thepatient's anatomy as well as one or more specified implants. Forexample, the surgeon may be able to choose from a catalog of implantshaving different physical characteristics such as size, shape, etc. Thesurgeon may choose the appropriate implant and manipulate the threedimensional rendition of the implant to fit over the three dimensionalrendition of the patient's anatomy in the desired alignment. The surgeoncan then select an option for the computer system to generate thesurgical plan comprising the planned resections required to prepare thepatient's anatomy to receive the implant. Accordingly, the computersystem may be configured to generate the appropriate mathematicalsurfaces to delineate the surgical plan by calculating the surfaces ateach intersection between the computer renditions of the implant and thepatient's anatomy as they have been aligned by the surgeon.

In order to guide the surgeon to follow the surgical plan with thesurgical instrument there must be a means for comparing the path of thesurgical instrument with the planned resection. The tracker-trackingelement subsystem may accordingly track the three dimensional locationand orientation of the mathematically defined surfaces of the surgicalplan relative to the tool. In one embodiment, the mathematical surfacesare referenced by the tracking element located at a fixed position onthe patient's anatomy. For better accuracy the tracking element may befixed to rigid tissue at an easily identifiable location. Doing so willsimplify registration of the patient's anatomy with the tracking systemand will avoid unwanted error that may be caused by unpredictablemovement of soft tissue. Once the patient's anatomy is registered withthe tracking system, the mathematical surfaces defined in the computersystem can be tracked based on their coordinates relative to coordinatesof the tracking element's fixed position. Since the tracking system islocated on the surgical instrument, tracking data collected by thetracking system regarding the location and orientation of the patient'sanatomy and the corresponding mathematical surfaces of the surgical planare relative to a defined reference point on the surgical instrument.Accordingly, during the surgery, the computer system may use thetracking data to make iterative calculations of the deviation betweenthe surgical path followed by the surgical instrument and the surfacesof the surgical plan. Errors in alignment between the surgical path andthe surgical plan as well as corrective actions may be communicated tothe surgeon by an indicator such as a graphical notification on acomputer screen, LCD, or projected display, a flashing light, an audiblealarm, a tactile feedback mechanism, or any other means for indicatingdeviation error.

In one aspect, an indicator is a system to provide guidance to thesurgeon on how to align the surgical path to achieve the intendedresection of the surgical plan. In one embodiment, the indicator is anelement of the computer system used to provide information to thesurgeon in the operating room. U.S. patent application Ser. No.11/927,429, at paragraph [0212] teaches the use of an operating roomcomputer to guide the surgeons operation of a surgical tool. One meansof indication taught in the '429 patent is the actuation of the surgicalinstrument. As the surgeon's surgical path deviates from the intendedresection, as detected by the on-board camera-tracking elementsubsystem, the computer system will communicate with the surgical toolto slow or even stop the tool from operating. In such a system, theactuation of the surgical tool is the means by which the surgeonreceives indication from the computer assisted surgery system as furthertaught in the '429 application at paragraph [0123].

In another embodiment, the computer system could indicate when thesurgical path deviates from the intended resection via an externaldisplay. The computer system can display a three dimensional renditionof the surgical tool and the patient's anatomy. Overlaid onto that imageis a three dimensional rendition of the surgical plan. The computersystem updates the relative position of the surgical tool and thepatient's anatomy, as determined by the camera-tracking element subsystem, and overlays the intended resections. The surgeon can thenutilize the display to align the surgical path with the intendedresection. Similarly, the relative position of the surgical tool and thepatient's anatomy can be displayed on other screens, such as a personaleyeware display, a large projected display in the operating room, asmartphone or a screen attached to the tool. The combination of anexternal screen, such as the one on the computer system, and otherscreens, such as a screen on the tool itself, may provide the surgeonwith an optimal amount of information. For example, the screen on thecomputer system can provide the surgeon with a global overview of theprocedure whereas the screen on the tool can provide particular guidancefor a specific resection or step in the procedure.

A screen on board the surgical tool is taught in the '429 application atparagraph [0215]. The on board screen could display the same kind ofimage as described above on external display. An exemplary implantationin the context of an OTT device is shown and described in FIGS. 52A and52B. The on board screen could display a simplified depiction of thealignment of the surgical path and the intended resection. In oneembodiment, the simplified display is comprised of three lines. Thesurgical path is depicted by two lines, one small and one large. Thesmall line depicts the distal end of the surgical path while the widerline depicts the proximal end of the surgical path. The third linedepicts the intended resection. The first two lines are calculated fromthe navigated position (location and orientation) of the surgical tool.The computer system compiles all three to display on the screen on thesurgical tool. The display shows both the proximal and distal parts ofthe surgical path, indicating to the surgeon its relative position inthree dimensions. When the surgical path is aligned with the intendedresection, all three lines are aligned. The indicator shows the surgeonhow to correct the position of the tool in three dimensions.

In one embodiment, the display is optimized to provide guidance fornavigating a saw. The surgical path is depicted by lines, which roughlycorrespond to the shape of the cut that a saw makes. In anotherembodiment, the simplified depiction could be depicted by two circles: asmall circle depicting the distal end of the surgical path and thelarger depicting the proximal end. A second shape that is roughlyequivalent in size, such as a cross or diamond, depicts the intendedresection. As previously described, the surgeon can align the surgicalpath to the intended resection by lining up the shapes. The circlesdepict the surgical path of a different tool, like a drill. In thismanner, the system can provide guidance for a wide variety of surgicaltools. In one embodiment, the position of all of the elements describedin the indicator should be updated, by the computer and tracking subsystems, at a rate that is faster than human reaction time.

One limitation of surgical displays is that they divert the surgeon'sattention away from the patient. One solution is to project theindication information directly onto the part of the patient's bodywhere the procedure is taking place. Any variety of projectors could beplaced onto the tool and display any of the indication methods onto thepatient. In one embodiment, an on board Pico projector could display thethree line simplified approach described above. In many respects, thethird line would be enormously helpful as it would depict, preciselyonto the patient, where the intended resection would start relative tothe rest of the patient's anatomy. In addition, the indicator canprovide more direct guidance as to how to correct the surgical path foralignment with the intended resection and project the guidanceinformation directly onto the patient. For example, the projector candepict an arrow that points in the direction the surgeon needs to moveto correct the surgical path.

There are several challenges to accurately project the indicationinformation onto the patient anatomy. Foremost, for an onboard,on-the-tool approach, the projection platform would be constantly inmotion. In addition, the surface that the projector is projecting on isnot flat. To resolve the second question the system utilizes informationobtained during the surgical planning. First, the system knows thegeometry of the surface of the patient's anatomy. The surgical plancontains a medical image of the patient, such as a CT scan, from whichit can extract the geometry of the surface that the indicator willproject on. The system accordingly projects guidance information so thatit is properly seen by the surgeon viewing the projected information onthe surface of the patient's anatomy For example, if the system is toindicate where the surgeon should cut with a saw, by utilizing astraight line, then the system can bend and curve the line so that, whenprojected onto the patient's anatomy, it will appear to be straight.Utilizing that approach, the indicator can project the three linesimplified depiction of alignment taught above.

Similarly, the system also calculates the relative position of the toolby means of the tracking system. With that information, the system cancontinuously modify the angle of projection to ensure that the indicatorprojects to the proper position of the intended resection on thepatient's anatomy. The indicator can use a wide variety of projectorssuch as a mini standard-LED projector or a laser-scanning pico projectorsystem. Notwithstanding, nothing in the forgoing prevents theutilization of a projector that is not on board the tool or used in anyother form of computer-assisted surgery. For example, an externallytracked system could include a separate projection system that wouldsimilarly project indication information onto the patient's anatomy.

In addition to a screen or a projector on board the saw, the system canutilize a smartphone or tablet computer, such as an Apple IPhone 4G, toprovide indication to the surgeon. An indicator that uses a smartphoneor tablet computer has the further advantage of a removable screen.Additionally, just as the on board screen, the smartphone can displayrenditions of both the tool and the patient or a simplified image, suchas the two line embodiment. A different simplified display could provideindication when the surgical path and the intended resection are alignedand direction when they are misaligned. For example, if the surgeon isapproaching the resection too low, then the screen can depict an arrowpointing up. The arrow can be rendered in three dimensions, providingfurther indication to the surgeon.

For simplified indicators, the display need not be as robust as asmartphone or other high-resolution screen. A bank of LED's, forexample, could display either the three line or arrow indicationpreviously described. The Indication method need not be visual. Thesystem could audibly indicate to the user when the surgical pathdeviates from the intended resection, as further described in the '429application at paragraph [0122].

As detailed above, computer assisted surgery proceeds from acomputer-based anatomical model such as those based on images andreconstruction obtained using any known medical imaging modality, orfrom anatomical models generated through morphing or other knownprocesses for rendering anatomical or bone models for use in computeraided surgery with the aid of computer-based anatomical models, asurgical plan is developed to be implemented for a specific patient andprocedure. Surgical preplanning includes a number of steps such asobtaining pre-surgery image data, surgical planning for the specificprocedure to be undertaken, adaptations of the plan for patient specificanatomy or condition and, if appropriate, to any specific prosthesis,devices, implants, or other structures to be placed in, joined to orused at a chosen 3D alignment during the CAS procedure. With thisgeneral pre-surgical planning information in hand the surgeon moves tothe patient specific intraoperative planning to be implemented at thesurgical site. The patient specific intraoperative surgical plan will beadapted to address the specific site or specific procedure such as anyorthopedic procedure or minimally invasive procedure that may beenhanced through the use of computer assisted surgery. For example aspecific joint may be aligned for some form of repair, for partialreplacement or for full replacement. It is to be appreciated that thetechniques described herein may be applied to other joints such as theankle, hip, elbow, shoulder or for other portions of the skeletalanatomy (e.g. osteotomies or spine surgery procedures) that wouldbenefit from the improvements to computer aided surgery describedherein. Examples of skeletal anatomy that may benefit from thesetechniques include, without limitation, vertebrae of the spine, theshoulder girdle, bones in the arm, bones in the leg, and bones in thefeet or hands.

By way of a non-limiting example a total knee arthroplasty will be usedas a specific example. For purposes of discussion the total kneearthroplasty will normally include five surgical cuts for the femur (ona CR or PCL retaining and eight cuts on a PS or PCL sacrificing) and oneor more cuts for the tibia each of them described below in greaterdetail. It is to be appreciated that these cuts may be modified toemphasize a particular aspect or aspects of a portion of a surgicalprocedure or step. For example, the specific geometry, orientation, orfeature of a prosthetic device for a particular procedure may lead tomodifications in certain aspects of the surgical plan. In anotherexample, a particular procedure or prosthesis may benefit from aspecific type of cut, tool, or surgical approach. Any of these factorsmay also be used to adjust the way that the computer aided surgeryproceeds according to the embodiments described herein. By way of anon-limiting example, the computer aided surgery system may select thesurface (e.g. plane) of cut as the most important information to bepresented to the surgeon immediately prior to or during a computer aidedsurgery step. In still further aspect, and OTT CAS will permit the userto select or base surgical step decisions using 2-D, 3-D or other outputinformation related to a representation of either the surgical toolbeing used or the resulting use of that tool on the anatomy. Forexample, if the surgical tool is a saw then the user may select fromrectangular shapes generally sized to correspond to the profile of thesaw, or to one or more surfaces (in this specific example a plane) thatcorrespond to the resulting cuts formed in the anatomy by the saw. In anadditional example, the surgical tool includes a drill and the user isprovided with or the system basis processing decisions using circlescorresponding to the size of the drill, cylinders related to theanatomical impact of the use of the drill, as well as other factors thatmight represent the engagement of the drill cutting tip to the anatomy.In still another example, the surgical tool includes a reamer or otherspherically shaped tool. In this example, the system or the user isprovided with circular, cylindrical, hemispherical, or sphericalrepresentations that are likewise used for display and feedback to theuser or as part of processing decisions used within the OTT CAS system.In a final example, the surgical tool includes a flat filing blade,whereby the representation will again be a flat surface (or thinrectangular block) depicting a certain thickness of filing action whichwould result upon contact to the anatomical surface.

In the embodiments that follow, an on-tool tracking system (OTT)embodiment is used to acquire, perform some data-processing on board,and provide real-time data regarding the surgical procedure to thecomputer-aided surgery computer, and to receive commands from the latterto set its own motor speed, attenuate speed or even stop to preventunintended cutting. The on tool tracking system is used to provide avariety of data for use by the computer aided surgery system. One formof data is imaging data from imaging sensors provided by the on-tooltracker. The data provided by these imaging sensors include for examplestereoscopic images, which once processed, can be used for tracking andinformation to be projected onto the surgical field by a standalone oran embodied projector or any type of projector provided for use with theon tool tracking system. Other data provided by the imaging sensorsincludes, reference frame location, orientation, alignment or otherphysical attribute of a reference frame used for defining the surgicalfield. One or more reference frames that may be positioned around thefield, around the joint, around the knee, or sized and shaped inrelation to a surgical field where the reference frame is visible duringat least a portion of all or substantially steps of a surgicalprocedure. (See, for example, reference frame embodiments described withregard to FIGS. 16-30. Still further, data may be selected only from arelevant reference frame or portion thereof based upon the dynamic, realtime assessment of a CAS procedure or CAS step.

For example, in a CAS procedure where two frames are present, both maybe used at the beginning of a cut and then the system shifts to usingonly one reference frame used during the cut. In a similar way, thesystem may use less than all the fiducial markers available on aspecific reference frame during a procedure in furtherance of the modeadjustments described below. Fewer fiducials to process may permitfaster updates or reduced image processing computer cycle time. As shownand described herein, the reference frames may have the same shape ordifferent shapes and may contain any of a variety of fiducial markers inany of a variety of suitable arrangement for detection by a visual or aninfrared tracking system in the OTT. Still further data available fromthe imaging sensors includes scene information such as anatomicalconfigurations of real or artificial anatomy or structures, markerspositioned on the patient, additional targets positioned around thesurgical field such as pointers, markers or the instrument being used inthe field such as a saw, drill, burr, file, scene information refers toimage capture, image processing or camera adjustments to select andprocess a portion of a frame, adjust a camera to zero in on or focus orzoom to a portion of interest in the surgical field based on real-timedynamic CAS procedures and consideration of a CAS surgical plan, reameror any other surgical tool to which the on tool tracking system ismounted.

When resecting the various portions it may be desirable to modify theview of the virtual model displayed on the OTT monitor. For instance,when cutting along a first plane it may be desirable to view the virtualmodel from a first perspective, and when cutting along a second plane itmay be desirable to view the virtual model from a second perspective.Accordingly, the OTT CAS system tracks various data regarding the statusof a procedure, including, but not limited to the following: theposition of the surgical tool relative to the tissue to be resected andthe orientation of the surgical tool relative to the tissue to beresected. Based on the position and orientation of both the tissue andthe surgical tool, the system calculates which surface is about to becut during the procedure and update the OTT monitor accordingly.

Further, the OTT CAS system can be configured to account for thepreference of each user as well as the characteristics of the instrumentusing the OTT device. Specifically, a surgeon may desire a differentview than the default view for a particular resection step or cuttingplane. The system allows the surgeon to override the default selectionand specify the view for a particular cut. The system stores theinformation regarding the desired view for the particular cut for theparticular surgeon and uses the view as the default view in the futurewhen the system determines that a similar cut is to be made. The systemtracks the user preference based on the user logged into the OTT CASsystem.

In addition to the types of data described above, the on tool trackingsystem may also provide other kinds of data such as output from one ormore sensors on the on tool tracker. Exemplary sensors include positionsensors, inclinometers, accelerometers, vibration sensors and othersensors that may be useful for monitoring, determining or compensatingfor movements of the tool that is carrying the on tool tracking system.For example, there may be sensors provided within the on tool trackingsystem to compensate for noises or vibrations generated by the tool sothat the noise and vibration may be compensated for i.e. cancel out ofthe imaging data or other OTT data being transmitted to the computeraided surgery system computer. In still another example, anaccelerometer or motion sensor may be provided to produce an output tothe computer aided surgery system used in predicting the next frame orestimating where relevant information in an imaging frame may be locatedbased on the movement of the tool and a tracking system. In stillanother aspect, sensors carried on board the on tool tracking system maybe used to detect, measure and aid in canceling unwanted movement thatmay interfere with, impair the quality of or complicate CAS or OTT imageprocessing. Specific examples of this type of feedback include sensorsto detect and aid in the cancellation of hand shaking or movement by theuser. In still another example sensors may be provided to detect and aidin the cancellation or compensation of unwanted movements or otherinterference generated during active surgical steps.

In other variations, image capture, processing and camera adjustment mayalso be used in or become the subject of compensation techniques,including to dynamically optimize the field-of-view andvolume-of-interest. In one example, a camera provided on the OTTcontains an auto focus capability that, under instructions from the CAScomputer and the various factors described herein, will dynamicallyadjust the camera and view to zoom, track, pan or focus on a frame, aportion of a frame or a natural or artificial feature. In anotheraspect, the imaging portion of a camera on the OTT is provided with asuitable on board movement system to tilt or adjust the lens to directthe lens to one or more features under the direction of the CAScomputer. This tilting lens may be used in conjunction with the dynamiclens above or with a lens having fixed (i.e., not adjustablecharacteristics). In one aspect, a micro mechanical base supporting thecamera lens is adjusted according to the instructions from the CAScomputer. It is to be appreciated that while the lens/camera adjustmentmay be done internally with a MEMS structure, it may be done external toas well. For example, a camera in a housing may be carried by a dynamicstage (x-y-z or x-y motion for example) where the state receiverinstructions from the CAS computer to adjust the camera position inaccord with the OTT CAS processes described herein. Still another formof compensation provides for image processing or other adjustments forOTT-tool orientation such as top mounted OTT, left side mounted OTT orright side mounted OTT. Still further, the various aspects describedabove for controlling the field of view (including either or both of thehorizontal and vertical field of view alone or in any combination) alongwith adjustments to a volume of interest within the surgical field maybe accomplished dynamically and optimized in real time utilizing theinstructions contained within the OTT CAS system, the CAS mode selectprocessing sequences and/or any of the specific CAS mode algorithmsincluding vision based algorithms or specific mode algorithms.

Another example of settings and compensation techniques include theimplementation and switching on/off of infrared filters placed in frontof the camera lens so that the imaging can be of infrared only oremitted or reflected by the reference frame markers to cut-out whitelight noise and to ease image processing and marker detection.

It is to be appreciated that these aspects of compensation may beimplemented mechanical components, electrical components or withsoftware, each alone or in any combination.

For purposes of discussion and not limitation the data from the on tooltracking system will be categorized as imaging data and sensor data tocapture the broad categories described above. Using system resourcesprovided either on the on tool tracking system itself or provided by thecomputer-aided surgery computer, the data is processed to provide anoutput for use by the computer aided surgery system. The desired outputof data processing comes in a number of different forms depending uponthe specific processes being evaluated and as described in greaterdetail below. For purposes of this overview, one may consider that thedata output obtained from the on tool tracking system may include suchthings as the orientation of the on tool trackers in the surgical field,the position of the tools or the on tool trackers in relation to thesurgical field, information regarding the surgical field such asphysical changes to the anatomy undergoing surgery, movement of the OTTtracked tool within the surgical field, displacement of the tool withinthe surgical field, apparent progress of the surgical step being trackedand other information related to the initiation, progress or completionof a surgical step or a computer aided surgical procedure.

The output of the on tool tracker, in whatever form suited to theparticular computer aided surgical procedure undertaken, is nextcompared to the step, or procedure undertaken according to the surgicalplan. The result of this comparison produces an output back to the ontool tracker that gives information related to the plan, step, orprogress with in a step of the surgical plan. In general, this output ismanifested for the user as the result of a projected image from aprojector on board the on tool tracker, but it can also include audiofeedback, changes/messages in a computer screen if available, actions onthe cutting tools (e.g. changes of cutting speed, direction andstopping), etc. It is to be appreciated that the output from thisprojector (as example) may be adapted based on a number ofconsiderations such as the available surgical field upon which an imagemay be projected, the likely position and orientation of the on tooltracker and its tool to the surgical field, and the likely challenges ofmaking the projected image visible to the user. As a result, the onboardprojector is capable of projecting images in a variety of configurationsbased upon the dynamic, real-time circumstances presented during thesurgical procedure. Moreover, the on tool tracking system may beprovided with additional illumination sources to enable the system orthe user to obtain image data in the visible spectrum, infraredspectrum, or in any other spectrum suited to image processing using theon tool tracking system. In still further aspects, one or more of theCAS mode processing methods described herein may be modified toincorporate the use of any of a variety of pattern recognition, computervision, or other computer-based tracking algorithms in order to trackthe location and orientation of the OTT instrument in space relative tothe surgical site, or relative to other instruments near the surgicalsite, and progress of an OTT CAS surgical step, without or substantiallywithout the use of reference frame-based tracking information. In otherwords, the embodiments of an OTT CAS method include the use of visualinformation obtained from the trackers or cameras on board the OTT forthe purpose of identifying, assessing, tracking, and otherwise providingthe CAS data sufficient for the purposes of providing appropriate CASoutputs for the user to complete one or more CAS processing steps. Inone aspect, a portion of the anatomy within the surgical field is markedor painted for the purpose of enhancing vision based tracking and visionbased algorithm processes. As a result of being provided informationfrom the projector of the on board tracking system, the user may respondto that information by making no change to his actions or by adjusting,as warranted under the circumstances for the step or procedure, one ormore of the operation, placement, orientation, speed, or position of thetool in the surgical field. The information from the projector may beprovided alone or in combination with other OTT components or feedbackor indications such as tactile or haptic feedback.

Next, the continued action or change of action by the user is detectedby the on tool tracking system and the process of providing dataprocessing data and providing it for comparison and evaluation by thecomputer aided surgical system continues.

Against this general overview is to be appreciated how, in use,embodiments of the on tool tracking enabled computer aided surgerysystem described in herein monitors and evaluates one or more of theposition, movement, use, predicted movement of an instrument using theon tool tracker against the planned computer aided surgery procedure andproduces appropriate computer aided surgery outputs to the user based atleast in part on a real-time computer aided surgery assessment by thecomputer aided surgery system.

Turning now from the general overview to more specific discussions ofhow computer aided surgery is modified by the use of the on tooltracking system described herein. FIG. 31A illustrates a general processflow of information for computer assisted surgery. FIG. 31B similarlyrepresents the general step wise approach used during the actualdelivery of the computer assisted surgical plan. These two flow chartswill be used to provide a general frame work for the improvement tocomputer assisted surgery according to embodiments described herein.

With reference to FIG. 31A, information obtained by the system isprocessed. This can include information from a variety of sourceslocated within the surgical field or from instruments used duringsurgical procedure in a continuously running feedback loop. Next, theinformation that has been obtained and processed is assessed using anappropriate computer assisted surgery algorithm. Finally, an output isproduced from the assessment to aid the user in performance of thesurgical procedure. The output produced may include one or more of thedisplay, a projected image, or an indication. Indications may include,for example, a tactile feedback signal including for example temperaturevariations, a haptic feedback signal with forces or vibration ofdifferent frequency and/or amplitude, remote or onboard control of theinstrument's motors or actuators with regards to their speed, direction,brake and stopping, an audio signal or visual signal provided to theuser in a manner appropriate to the circumstances and use of the on tooltracking system and the instrument attached thereto.

While similar to the conventional computer aided surgery in somerespects, the systems and techniques described herein are different andprovide unique advantages over conventional computer assisted surgerysystems and methods.

The on tool image and projection module is adapted and configured with anumber of different characteristics based upon the type of computerassisted surgery being undertaken. OTT position in relation to surgicalfield during expected use for a CAS procedure, orientation of projectorto the tool being guided, shape and surface condition (i.e., roughpresence of blood or surgical debris) of the surface in the surgicalfield being projected on, horizontal field of view accommodation,vertical field of view accommodation are just a number of theconsiderations employed in the embodiments described herein.

Still other embodiments of the computer aided surgery system describedherein compensate for variations and alternatives to the componentselection and configurations resulting from the above describedfeatures. One exemplary compensation relates to camera adjustment orimage adjustment (discussed above) for the surgical step or fieldadjustment based on a particular computer aided surgery technique.Another exemplary compensation relates to the actual projector positionon a particular embodiment. The projector position of a particularembodiment may not be on the centerline of the device or in an optimumposition based on horizontal or vertical field of view or may be tiltedin order to address other design considerations such as making a devicesmaller or to accommodate other device components. One form ofcompensation for this aspect is for the projector output to be adjustedbased on the actual projector location. This type of compensation issimilar to keystone adjustments for a projector output. The projectorprovided on board the on tool tracking system may have its outputcompensated for the expected or actual portion of the surgical fieldwhere the projector output will display. During the surgical procedurethe surgical site is likely not to be flat and so would not faithfullyreflect the intended image from the projector. However, since thegeometry of the target anatomy (e.g. bone surface) is known, the imageto be projected by the projector can be changed by software tocompensate such that when projected on the non-flat surface, it wouldappear clearer as intended to the user. The target anatomy surface forprojection may vary in shape, orientation, curvature or presence ofdebris, blood and still further, the output of the OTT projector may beadjusted based on real time factors such as these detected by the OTTvision system and object detection techniques. When the cutting hasstarted, there would be a new source of ‘un-flatness’, namely, theinterface between the original native surface of the bone, and the newsurface introduced by the cut. This can be calculated (and compensatedfor) during cutting by logging where the cut was made, or assumed to bethe desired ideal/planned surface, or digitized (e.g. with the pointer)after each cut.

Still further differences between the OTT surgical technique andconventional computer assisted surgical techniques include the types andmanner of providing outputs or receiving inputs from the on tooltracking system or the user. Sensors and systems to provide tactile,haptic or motion feedback may be used as well as a variety of indicatorssuch as alarms, visual indicators or other user inputs specific to thecapabilities of a specific OTT system.

FIG. 31B relates the general OTT enabled CAS process with added detailsto call of additional aspects of the OTT CAS system. When the procedurebegins, the user has a selected surgical tool with the on tool trackingsystem mounted thereto in either top mount, right side mount, left sidemount or bottom mount as determined by the user and the OTT CAS plan.The tool with attached OTT is identified to the system through a toolregistration procedure such as the tool transmitting an identificationsignal or a self-registration process or other suitable registrationprocess. The pre-surgical planning steps, as needed, are completedaccording to the procedure to be undertaken. Beginning with the computeraided surgery surgical plan, the user initiates a computer aided surgerystep. As a result of the use of the on tool tracking system, on tooltracking data is generated. The on tool tracking data is processed andthen provided to the computer system that compares and assesses theplanned surgical step information to that received from the on tooltracking data. As a result of this comparison and assessment of the ontool tracking data, an appropriate output is provided to the user or tothe OTT's on board motor control circuitry as a motor or actuatorcontrol signal to slow, stop or reverse the instrument or let itcontinue at the speed desired by the user through the manual onboardhand trigger. This output is detected and acted upon by the on tooltracking system which provides additional data that is again provided tothe tracking computer. Next the user responds to the output provided andeither continues the current action, or changes the use of the toolbeing tracked by the on tool tracking system. The users response,whether involving action or not, is detected by the on tool tracking andbecomes additional data input to the surgical computer. These processescontinue as the computer system processes the progress of the stepagainst the surgical plan. If the answer to step completion is no,comparison of data and output to the user continues. If the answer tostep completion if yes, then the user may initiate the next surgicalstep or the surgical planning computer may provide an output to the userto notify him that one step is completed and any one of other remainingother steps can be undertaken. The sequence of CAS steps to be performedare totally up to the user, except in situations where one step cannotbe performed without a prerequisite other step(s) identified in the setsurgical plan. The control is totally in the hands of the user, with thecomputer being only (optionally) suggestive of what steps can be done,or (optionally) prohibitive of what steps cannot be done. Theseprocesses continue in accordance with computer aided surgery proceduresuntil the plan is delivered. If the plan is complete, the use maydetermine whether any real-time revision of the surgical area is to beundertaken. The revision process may also be tracked and monitored toprovide information to the user. If no revision is required or the CASplan is completed, then the CAS plan is completed.

FIG. 32 provides a flowchart that will be used to describe still anotherimprovement to computer aided surgery provided by embodiments of the ontool tracking system described herein. As before, the system willcollect and process computer aided surgery data. Next, the computeraided surgery system will assess the CAS data during the CAS procedure.As a result of this assessment, the CAS computer will determine the CASprocessing mode. Thereafter, mode based processed adaptation will beapplied to the data used in the CAS process. Finally, the OTT CAS systemprovides a user or the instrument motor/actuator a CAS output (or speedand motor direction set-point) based on the processing mode.

Mode selection relates to the OTT CAS system ability for a dynamic, realtime assessment and trade off of a number of aspects of the CASoperation including the need to update the user, processing rates,cutting instrument motor control/actuation instantaneous speed andprospective response times and requirements to obtain improved ordifferent data, relative importance of portions of data based upon CASstep progress or interaction with the patient or other factors relatingto the overall responsiveness of the OTT CAS system. Additional aspectsof the step of determining the CAS processing mode described above inFIG. 32 may be appreciated with reference to FIG. 33. FIG. 33 relates tothe inputs considered by the system to determine the processing mode andthe result of that determination. Exemplary inputs used by the OTT CASsystem for determining processing mode include, by way of example andnot limitation, one or more of the following: speed or motion of thetool or its motor/actuator speed, input or indication from a toolmonitoring device, voice input or indication from user, physicalparameters in the surgical field, including natural or artificialparameters; reference frame input; projected image; motion detectionfrom sensors; motion detection from calculations; overall CAS procedurestatus; CAS step status; user input (e.g. CAS screen, OTT touch screen,touch screen, motions sensor, gesture recognition, GUI interface, etc.);CAS step progress including, for example, percentage complete,deviations from plan, real-time adjustments. As a result of thedetermination step performed by the OTT CAS computer a processing modewill be selected based on the real-time circumstances and evaluation ofthe surgical procedure as made by the algorithms of the CAS for OTTcomputer. Criteria used by the OTT CAS computer for determining modeinclude such factors as the physical proximity of the surgical tool tothe patient anatomy, actions being undertaken by the user, sensor inputsof tool motion, predicted tool motion, speed of tool motion, speed ofthe tool's motor or cutting actuator and other factors related to theplacement, orientation, or use of a surgical tool within the OTT imagefield. By way of non-limiting example, CAS processing modes may includea hover mode, a site approach mode, and an active step mode. In generalterms, hover mode refers to those circumstances during an OTT CASprocedure when the on tool tracker and tool is near or within thesurgical field without contact between the tool and the patient. Ingeneral terms, site approach mode refers to those circumstances duringan OTT CAS procedure when the on tool tracker and tool is within thesurgical field and in contact with patient, but without the toolactively engaging the patient anatomy to perform a surgical step such assawing, cutting, reaming, drilling, burring, shaving, filing and thelike. In general terms, active step mode refers to those circumstancesduring an OTT CAS procedure when the on tool tracker and tool is engagedwith the patient anatomy to perform a surgical step such as sawing,cutting, reaming, drilling, burring, shaving, filing and the like. As aresult of the determine CAS processing mode decision, the OTT CAScomputer will adapt the CAS processing mode to or between: hover mode,site approach mode, or active step mode as is appropriate under thecircumstances.

Step of adapting the CAS process to a particular mode as described abovewith regard to FIG. 33 is further described with reference to FIG. 34.In general terms, the OTT CAS computer is adapted and configured toadapt the CAS process mode based on adjustment factors to produce aparticular mode processing algorithms. By way of example, the variousmode adjust processing factors are shown in FIG. 34. Based on theprocessing inputs as detailed in the flowcharts above, the OTT CAScomputer will adjust the processing steps undertaken for OTT CAS basedon one or more of or combinations of or variations of the following CASmode processing adjustment factors: camera frame size and/or cameraorientation (if camera software or firmware provides for suchadjustment); adjustments to camera image outputs to modify a size of aregion of interest within a horizontal field of view, the vertical fieldof view or both the horizontal and the vertical field of view of thecamera; drive signals for adjustable camera lens adjustment orpositioning; image frame rate; image output quality; refresh rate; framegrabber rate; reference frame two; reference frame one; on referenceframe fiducial select; off reference frame fiducial select; visualspectrum processing; IR spectrum processing; reflective spectrumprocessing; LED or illumination spectrum processing; surgical toolmotor/actuator speed and direction, overall CAS procedure progress;specific CAS step progress; image data array modification; picoprojector refresh rate; pico projector accuracy; set projector or otherOTT electronics “OFF” or in sleep mode or power save mode; imagesegmentation techniques; logic-based extraction of an image portionbased on a CAS progress; signal-to-noise ratio adjustment; imageamplification and filtering; weighted averages or other factors fordynamic, real-time enhancement or reduction of imager rate, pixel orsub-pixel vision processing; hand tremor compensation; instrument-basednoise compensation (i.e. saw vibration compensation). Put another way,the various factors listed above may be grouped into the various ways ofproviding adjustments of the camera based on those adjustments that cantake place within the camera such as in the software or firmware oroperating modalities provided by the camera electronics themselves onthe one hand. And on the other hand, on a broader scale, the overalladjustment of the camera in its housing in relation to the OTT housing.In this way camera movement speaks of a more general shifting of theentire camera body or the camera lens itself rather than internalelectronic modifications or adaptations of camera output based onelectronic processing of camera image information. For within cameravariations these are such things as focal point, zoom, exposure,aperture and other camera based modifications that will adjust thecameras output as part of an imaging adjustment. In one specificexample, one or more of the above features are used to produce a hovermode CAS algorithm that is used during hover mode processing adaptation.In one specific example, one or more of the above features are used toproduce an approach mode CAS algorithm that is used during approach modeprocessing adaptation. In one specific example, one or more of the abovefeatures are used to produce an active step mode CAS algorithm that isused during active step mode processing adaptation.

FIG. 35 illustrates a flowchart of an exemplary OTT CAS process buildingupon the steps described above. Collect and process CAS data. Assess CASdata during a CAS procedure. Determine CAS processing mode. Undertakemode based CAS assess adaptation. Based on the result of the mode baseddetermination, if hover mode, apply hover mode CAS algorithm toprocessing. Provide the user with hover mode CAS outputs, or provide theOTT motor control circuitry with speed control commands/signals.Exemplary user outputs include hover mode display outputs, hover modeprojected image outputs, hover mode indications such as tactile, haptic,audio and visual indications adapted to the processing steps used in thehover mode. Based on the result of the mode based determination, if siteapproach mode, apply site approach mode CAS algorithm to processing.Provide the user with site approach mode CAS outputs. Exemplary outputsinclude approach mode display outputs, approach mode projected imageoutputs, approach mode indications such as tactile, haptic, audio andvisual indications adapted to the processing steps used in the approachsite mode.

Based on the result of the mode based determination, if active stepmode, apply active step mode CAS algorithm to processing. Provide theuser with active step mode CAS outputs. Exemplary outputs include activestep mode display outputs, active step mode projected image outputs,active step mode indications such as tactile, haptic, audio and visualindications adapted to the processing steps used in the active stepmode.

FIG. 36 illustrates a flowchart amid exemplary OTT CAS process basedupon those described above but using a unique trigger action indicatortool monitor or tactile or haptic feedback to further provide benefitsto users of an OTT CAS system. Various alternative embodiments of thetrigger action indicator are provided below with regard to FIGS.37A-52B. As before, the OTT CAS process proceeds by collecting andprocessing CAS data. In one alternative aspect, the collection andprocessing may also include an indication from the trigger action. Next,following the processes described above, the OTT CAS system will assessCAS data during a CAS procedure. Here again, a trigger action indicationmay also be applied to this step and assessed along with other CAS data.Thereafter, the user will be provided with an appropriate CAS outputbased upon the use of one or more trigger action indicators as describedabove. The appropriate CAS outputs may include a display, a projectedimage, or any of a number of indications such as tactile indications,haptic indications, audio indications or visual indications as describedabove or as are typical in CAS procedures.

Against this backdrop of the various aspects of OTT CAS processes, thefollowing examples are provided.

It is to be appreciated that OTT CAS mode may be detected and determinedby many factors (e.g., reference frame(s), positions, relative motion,etc.). Additionally, in the context of a surgical procedure, there isalso benefit in relating the defining attributes of an OTT CAS modebased on tool/target proximity or use. Consider the following examplesof: A) Hover: both tool and target within surgical field, but nocontact; B) Approach: Both tool and target within surgical field ANDthey are in contact; and C) Active step mode: Both tool and targetwithin surgical field AND they are in contact AND there is activeengagement of tool with tissue. In one aspect, the OTT deviceelectronics incorporates this mode selection functionality in a ‘smartviews’ module. This module is provided within the main CAS systemcomputer or within the OTT device where electronics including softwareand firmware implement all or a substantial part of the modes detectionalgorithms, and triggers the different events of the OTT CAS modeselection functionality.

In some additional aspects of OTT CAS mode control, one or more of thefollowing variations or alternatives may be incorporated:

-   -   1. Due to the temporal/special resolution on an OTT CAS system        and CAS system generally, some embodiments of the Approach mode        may be considered appropriate when tool and target are within a        given user-pre-selected (settable) distance envelope. The        distance envelope may be designated in a measurement range. One        exemplary range may be between 10 mm to 0 mm as determined by        the OTT CAS system. In other aspects, the Approach mode may be        delineated by the OTT CAS system determining that there is        likely contact between an active element of a surgical tool and        the anatomy within the OTT CAS surgical field.    -   2. In some aspects, an OTT CAS mode is provided with a        ‘hysteresis’ factor. This OTT CAS hysteresis factor is selected        to include the types of circumstances or CAS conditions that, if        satisfied such as continuously for a pre-determined time period,        will result in that CAS mode being maintained. In other words,        the parameters of the OTT CAS mode hysteresis must be met        continuously during a period of time to ‘lock into the mode’ or        maintain that OTT CAS mode. As used herein, continuously is        meant to be within the context of the time domains of OTT        processing times and sample rates and is not intended to denote        the absolute non-interruption of the conditions monitored. By        way of similar example, the hysteresis or some of the hysteresis        conditions have to NOT be met continuously during a period of        time to ‘un-lock’ or permit adjustment of the OTT CAS mode. The        use of OTT CAS mode hysteresis factors improves the system        response to transients, avoids or reduces the likelihood of the        system to jump from one OTT CAS mode to another inappropriately        and improves usability of the system since the user is likely to        see more stable OTT CAS outputs as the system will be providing        those outputs from a single OTT CAS mode.    -   3. During some OTT CAS steps, there are activities performed by        the user that may not require use of the projector, may require        different input-output (IO) devices (e.g. during implant        location assessment it may not be possible to project        information on the bone), and/or may not have a defined        target-tool relationship (e.g. knee range of motion assessment        only requires seeing tibial and femoral reference frames). It is        to be appreciated that the OTT CAS system may also receive        inputs from other sources and there are OTT CAS outputs where no        projector output is provided or utilized.    -   4. In general, the processing algortihms and OTT CAS mode        factors are selected based on the probability or likelihood        that, as for such things as the relative motion for bones,        instruments, implants, etc. will be decreasing as the OTT CAS        mode progresses from Hover to Active. The one exception to this        general process assumption is when the OTT CAS device or system        is used for the process of an assessment of a range of motion        for an involved joint within the surgical field or for that        joint that is the objective of the OTT CAS procedure or step.

OTT CAS Mode Examples

Bone Registration:

Objective: Finding out the geometrical relation between the origin ofthe reference frame and the origin of the bone model.

Procedure: Digitization of points on the surface of the bone with a tool(e.g. navigated pointer), and processing of these points againstpre-determined geometry data of the bone model

How the OTT CAS System Identifies this Task:

-   -   Pointer's AND bone's (either tibia or femur) reference frames        (RFs) are visible to OTT.

Initiation of the Task:

-   -   The OTT CAS system recognizes both reference frames coexisting        in the scene (for at least a minimum period of time suited for        this registration)    -   An additional ‘guess’ factor is the stage of the procedure        because for example, cutting cannot be done until the bones are        registered.) In this case, the trigger for this event may be the        OTT device is maintained in position to keep two reference        frames within the field of view until a bone registration        process is completed. This trigger can optionally be confirmed        by the system computer prompting the user to confirm and they        respond.    -   The information obtained during OTT device bone registration may        be annotated or overwritten if needed by user's input (touch        screen, voice command, touching with the pointer on a specific        divot on the bone's reference frame, etc.)    -   The latter (divot) is a specified point (position) on the        reference frame that when touched by a navigated pointer, would        tell the system that the user is intending to perform a task (or        one of the dedicated tasks) which involve that reference frame        itself. For example, this could be a registration of the bone        attached to that reference frame, and this may also invoke a        change of mode from eg. from Hovering/smart-views to        registration screen etc.

OTT CAS Modes

Hovering:

-   -   Range Condition: OTT device is too far away from the RFs, or the        2 RFs are too far apart. The range to trigger this condition is        settable during the calibration/tuning of the system, or by user        preferences, and is specified as a distance threshold between        the cameras to the target anatomy reference frame beyond the        optimum FOV (in our embodied case greater than 200 mm).    -   Tracker: Lower refreshing rate    -   Projector: May not project any image on the bone (as the bone        location is not yet defined), but can project elementary helpful        information such as confirming this mode/status etc. on any        reflective surface which happens to be in the way. Low        refreshing rate, limited by the trackers.    -   System: Monitors the pointer's tip and the bone's RF location in        ‘world’ coordinates. Drives tracker, projector, and other IO        devices.

Approach:

-   -   Range Condition: Medium OTT/RFs and RF/RF distances. The range        to trigger this condition is settable during the        calibration/tuning of the system, or by user preferences, and is        specified as a distance range from the target anatomy reference        frame such as 100-200 mm.    -   Tracker: High refreshing rate, optimizing pointer and bone RFs        readings (e.g. ignoring or disregarding other RF's)    -   Projector: As above, may not project any defined image (as the        bone location is not yet defined), but can project a solid        screen that changes colors (e.g. red, yellow and green) based on        ‘readiness’ to start collecting registration points.    -   System: Monitors the pointer's tip and the bone's RF location in        ‘world’ coordinates. Drives tracker, projector, and other IO        devices.

Active:

-   -   Smaller OTT/RFs and RF/RF distances. For example, less than        70-100 mm distance from the target reference frame, again        settable by user preferences as above.    -   Tracker: High refreshing rate, optimizing pointer and bone RFs        readings    -   Projector: As above.    -   System: Monitors the pointer's tip and the bone's RF location in        ‘world’ coordinates. Records pointer's tip location for each        digitized bone. Drives tracker, projector, and other IO devices.        Monitors progress of the registration process, and when finished        it calculates the final registration matrix.    -   May or may not require additional IO device (e.g. touch screen)

OTT CAS Considerations for Transitions Between Modes:

-   -   Mode shift is based on distance thresholds.    -   If there is no bone registration information then it is not        possible to determine bone-pointer ‘contact’ or ‘closeness’. The        system alternatively looks at a nominal distance between the        pointer (which IS registered) and the bone's reference frame        (instead of the bone itself). The resulting nominal distance may        then be used to estimate or assume approximate registration        based on the nominal position in which that (bone) reference        frame is usually recommended to be placed (see picture sheet        18-23). Another alternative is to (optionally) simply use any        old registration information by the system (of another default        bone or one from a previous patient or surgery) to make the        approximate registration for the purposes of determining what        “mode” the system should be in. The availability of this option        is also settable/selectable by the user.    -   Or by user's input.

End of the Task:

-   -   All registration landmarks have been visited and pointed        (registration process is fully completed).    -   OR the system ceases to see the pointer's RFs (for at least a        minimum period of time)    -   Alternatively, the process could be complemented or overwritten        by user's input (touch screen, voice command, touching with the        pointer on a specific divot on the bone's reference frame, etc.)

Bone Cutting/Drilling:

-   -   Objective: Re-shaping the bone with a tool (usually a powered,        smart instrument such as a saw, drill, burr, file, etc.) to        allocate and implant.    -   Procedure: Following the system's direction, the user        cuts/drills (usually) one surface at a time. This particular        activity applies to different individual ‘target surfaces’ on        each bone, one per cut/hole to be performed, so the system will        maintain such reference when using or processing locational or        orientational errors of the tool relative to the bone. Different        tools have different active elements (e.g. cutting tips), and so        the different active elements of each tool shapes result in        different 2D and 3D modification of the anatomy when the tool or        tool active element interacts with the anatomy in the surgical        field. As such, the guidance for each tool will vary with the        type of tool and active elements in use during an OTT CAS        process step.

How the System OTT CAS System Identifies this Task:

-   -   OTT detects at least one bone's reference frame (RFs).    -   The named bone is registered.    -   The reference frame of the bone being cut is within a user        selectable maximum distance (say, for example only, less than        200 mm).

Initiation of the Task:

-   -   The system recognizes both RFs coexisting in the scene (for at        least a minimum period of time)    -   This could be complemented or overwritten by user's input (touch        screen, voice command, touching with the pointer or the cutting        instrument on a specific divot or mark on the bone's reference        frame or the bone itself, etc.)

Modes

Hovering:

-   -   OTT is too far away from the bone. For example, more than 200 mm        (values settable by the user).    -   Tracker: Lower refreshing rate    -   Projector: May not project any image (the bone could be out of        the projector's sight) or may just display rough shapes (e.g.        arrows to indicate in what direction to move the instrument—e.g.        saw, drill, etc.—to align it with the bone). Optionally, the        projector output is modified to simply show different colors as        in the previous example. Low refreshing rate, limited by the        tracker's refresh settings.    -   System: Monitors the tool location and orientation relative to        the bone (i.e. in bone's coordinates). Drives tracker,        projector, and other IO devices. Communicates bi-directionally        and drives smart instruments.

Approach:

-   -   OTT is at medium distance to the bone. For example, between 100        mm and 200 mm.    -   Tracker: High refreshing rate, optimizing pointer and bone RFs        readings.    -   Projector: Shows alignment aids (colored text, lines, circles,        arrows, etc.) corrected for bone geometry at medium refreshing        rate.    -   System: Monitors the tool location relative to the bone (i.e. in        bone's coordinates) and calculates roll, pitch, yaw, and        distances deviations. Drives tracker, projector, and other IO        devices. Communicates bi-directionally and drives smart        instruments.

Active:

-   -   OTT is close to the bone. For example, between 70 mm and 100 mm.    -   Tracker: High refreshing rate, optimizing pointer and bone RFs        readings.    -   Projector: Shows alignment aids (colored text, lines, circles,        arrows, etc.) corrected for bone geometry at high refreshing        rate.    -   System: Monitors the tool location relative to the bone (i.e. in        bone's coordinates) and calculates roll, pitch, yaw, and        distances deviations. Drives tracker, projector, and other IO        devices. Communicates bi-directionally and drives smart        instruments at higher speed.

Transition Between Modes:

-   -   Transition may be based on distance thresholds.    -   Transition based on user input.

End of the Task:

-   -   User moves on to another task    -   All cuts and refinements are fully completed.    -   In one alternative, the OTT CAS system ceases to see the bone's        RFs (for at least a minimum period of time)    -   This step could be amended, complemented or overwritten by        user's input (touch screen, voice command, touching with the        pointer on a specific divot on the bone's reference frame, etc.)

Assessment of Bone Cut:

-   -   Objective: Evaluating a new surface (e.g. plane, cylindrical        hole, etc.) orientation, surface roughness, depth, etc.    -   Procedure: Total or partial digitization of the surface (e.g.        touching/traversing it with a navigated pointer), assessing a        cut location and orientation with a ‘surface monitor’ (a        navigated tool with a flat surface that sits on the flat cut),        gaging the depth of a hole with a navigated pointer, etc.

How the OTT CAS System Identifies this Task:

-   -   OTT sees at least one bone's reference frame (RFs) as well as        the assessing instrument's (surface monitor or pointer) RF.    -   The named bone and the instrument have been registered.    -   At least a cut has been performed.    -   The bone being cut is within a maximum distance ‘D’.

Initiation of the Task:

-   -   The system recognizes both RFs (bone and instrument) coexisting        in the scene (for at least a minimum period of time), while the        conditions above are fulfilled.    -   This could be complemented or overwritten by user's input (touch        screen, voice command, touching with the pointer or the cutting        instrument on a specific divot or mark on the bone's reference        frame or the bone itself, etc.)

Modes

Hovering:

-   -   OTT is too far away from the RFs, or the 2 RFs are too far        apart.    -   Tracker: Lower refreshing rate.    -   Projector: May not project any defined image (as the bone can be        out of projector's sight), or it can project a solid screen that        changes colors (e.g. red, yellow and green) based on ‘readiness’        to start the process. Low refreshing rate, limited by the        tracker's.    -   System: Monitors the tool location relative to the bone (i.e. in        bone's coordinates). Drives tracker, projector, and other IO        devices.

Approach:

-   -   OTT is at medium distance to both RFs AND medium bone-tool        distance.    -   Tracker: High refreshing rate, optimized for instrument and bone        RFs readings.    -   Projector: May not project any defined image (as the bone can be        out of projector's sight), or it can project a solid screen that        changes based on ‘readiness’ to start the process. Medium        refreshing rate.    -   System: Monitors the tool location relative to the bone (i.e. in        bone's coordinates). Drives tracker, projector, and other IO        devices.

Active:

-   -   OTT is at medium/close distance to both RFs AND small bone-tool        distance.    -   Tracker: High refreshing rate, optimized for instrument and bone        RFs readings.    -   Projector: May not project any defined image (as the bone can be        out of projector's sight), or it can project a solid screen that        changes based on process status (start to end of data        collection). High refreshing rate.    -   System: Monitors the tool location relative to the bone (i.e. in        bone's coordinates). Records pointer's tip location for each        digitized point or surface monitor location and orientation.        Drives tracker, projector, and other IO devices. Monitors        progress of the assessment process, and when finished it        calculates, records and displays the calculated parameters.    -   May or may not require additional IO device (e.g. touch screen)

Transition Between Modes:

-   -   Simply based on distance thresholds.    -   Or by user's input

End of the Task:

-   -   Assessment process is fully completed.    -   Optionally, the OTT CAS system ceases to see the instrument's        RFs (for at least a minimum period of time)    -   This could be complemented or overwritten by user's input (touch        screen, voice command, touching with the pointer on a specific        divot on the bone's reference frame, etc.)

Assessment of Implant Fit and Alignment

-   -   Objective: Comparing the actual location of the implant (or        trial) on a bone, relative to where it was expected to be        according to plan. This can happen during trial, and        before/during/after implant cementing or locking.    -   Procedure: An implant (e.g. femoral component, tibial tray,        etc.) gets a RF attached, and is tracked in ‘bone’ coordinate        system. At any given time the system can display/record its        position (relative to the bone), and instant errors (if any)        compared to where it was supposed to be.

How the System Identify this Task:

-   -   OTT sees at least one bone's reference frame (RFs) as well as        the corresponding implant's RF.    -   The named bone and the implant have been registered.    -   All cuts have been performed.    -   The bone being and implant are within a maximum distance ‘D’.

Initiation of the Task:

-   -   The system recognizes both RFs (bone and implant) coexisting in        the scene (for at least a minimum period of time), while the        conditions above are fulfilled.    -   This could be complemented or overwritten by user's input (touch        screen, voice command, touching with the pointer or the cutting        instrument on a specific divot or mark on the bone's reference        frame or the bone itself, etc.)

Modes

Hovering:

-   -   OTT is too far away from the RFs, or the 2 RFs are too far        apart.    -   Tracker: Lower refreshing rate.    -   Projector: May not project any defined image (as the bone can be        out of projector's sight), or it can project a solid screen that        changes colors (e.g. red, yellow and green) based on ‘readiness’        to start the process. Low refreshing rate, limited by the        tracker's.    -   System: Monitors the implant/trial location relative to the bone        (i.e. in bone's coordinates). Drives tracker, projector, and        other IO devices.

Approach:

-   -   Medium OTT/RFs distance AND implant/trial relatively close to        the bone.    -   Tracker: High refreshing rate, optimized for implant/trial and        bone RFs readings.    -   Projector: May not project any defined image (as the bone can be        out of projector's sight), or it can project a solid screen that        changes based on ‘readiness’ to start the process. Medium        refreshing rate.    -   System: Monitors the implant location relative to the bone (i.e.        in bone's coordinates). Drives tracker, projector, and other IO        devices.

Active:

-   -   Smaller OTT/RFs distance AND implant/trial is close/touching to        the bone.    -   Tracker: High refreshing rate, optimized for implant and bone        RFs readings.    -   Projector: May not project any defined image (as the bone can be        out of projector's sight), or it can project a solid screen that        changes based on process status (start to end of data        collection). High refreshing rate.    -   System: Monitors the implant/trial location relative to the bone        (i.e. in bone's coordinates). Calculates and displays (and        record when needed) the errors defined by the actual        location/orientation of the navigated implant relative to where        it is supposed to be according to plan. Drives tracker,        projector, and other IO devices. Monitors progress of the        assessment process, and when finished it calculates, records and        displays the calculated parameters.    -   May or may not require additional 10 device (e.g. touch screen)

Transition Between Modes:

-   -   Simply based on distance thresholds.    -   Or by user's input

End of the Task:

-   -   Assessment process is fully completed.    -   (or) The system ceases to see the instrument's RFs (for at least        a minimum period of time)    -   This could be complemented or overwritten by user's input (touch        screen, voice command, touching with the pointer on a specific        divot on the bone's reference frame, etc.)

Range of Motion:

-   -   Objective: Assess the range of motion and biomechanics of the        joint after implantation. It can be done with trials or final        implants on.    -   Procedure: After placing the trial (or actual implant) on,        before removing the bones' RFs and closing the wound, the        surgeon flexes the knee and performs handles the joint, reaching        limit positions like maximum flexion and hyper extension). This        maneuvering is performed while pointing OTT to the tibial and        femoral RFs. Dynamic measurements (tibia relative to femur) are        expressed in anatomical terms.

How the System Identify this Task:

-   -   OTT sees both tibia's and femur's reference frames (RFs).    -   Both bones have been cut. (Bone cutting and implant location        could have or could have not been performed.)

Initiation of the Task:

-   -   The system recognizes both RFs coexisting in the scene (for at        least a minimum period of time), while the conditions above are        fulfilled.    -   This could be complemented or overwritten by user's input (touch        screen, voice command, touching with the pointer or the cutting        instrument on a specific divot or mark on the bone's reference        frame or the bone itself, etc.)

Modes

Hovering:

-   -   OTT is too far away from the RFs.    -   Tracker: Lower refreshing rate.    -   Projector: May not project any defined image (as the bone can be        out of projector's sight), or it can project a solid screen that        changes colors (e.g. red, yellow and green) based on ‘readiness’        to start the process. Low refreshing rate, limited by the        tracker's.    -   System: Monitors the tibia location relative to the femur.        Drives tracker, projector, and other IO devices.

Approach:

-   -   Medium OTT/RFs distance.    -   Tracker: High refreshing rate, optimized for bones' RFs        readings.    -   Projector: May not project any defined image (as the bone can be        out of projector's sight), or it can project a solid screen that        changes based on ‘readiness’ to start the process. Medium        refreshing rate.    -   System: Monitors the implant location relative to the bone (i.e.        in bone's coordinates). Drives tracker, projector, and other IO        devices.

Active:

-   -   Smaller OTT/RFs distance AND implant/trial is close/touching to        the bone.    -   Tracker: High refreshing rate, optimized for implant and bone        RFs readings.    -   Projector: May not project any defined image (as the bone can be        out of projector's sight), or it can project a solid screen that        changes based on process status (start to end of data        collection). High refreshing rate.    -   System: Monitors the tibia location relative to the femur.        Calculates and displays (and record when needed) the dynamic        motion (flexion/extension, varus/valgus, internal/external        rotation, AP motion, etc.). Drives tracker, projector, and other        IO devices. Monitors progress of the assessment process, and        when finished it saves all parameter recorded and notifies the        user.    -   May or may not require additional IO device (e.g. touch screen)

Transition Between Modes:

-   -   Simply based on distance thresholds.    -   Or by user's input

End of the Task:

-   -   Assessment process is fully completed.    -   (or) The system ceases to see the bones' RFs (for at least a        minimum period of time)    -   This could be complemented or overwritten by user's input (touch        screen, voice command, touching with the pointer on a specific        divot on the bone's reference frame, etc.)

Other activities (e.g. registration verification, bone cut refinement,etc.) can be considered sub-cases of the above.

In one aspect in any of the above described examples, lower refreshingrate refers to changes in refresh rate from about 30-100 Hz to as low as1-10 Hz.

When resecting a portion of a bone a surgeon may cut more rapidly andaggressively when the cutting tool is relatively far from the boundaryof the area to be resected. As the OTT CAS detects the surgeonapproaching the boundary of the resection area, the surgeon may receiveappropriate OTT CAS outputs to slow the pace of cutting to ensure thatthe resection remains within the desired boundaries. To help the surgeonreadily assess the proximity to the resection boundary, the OTT CASsystem may provide a number of appropriate OTT CAS outputs to thesurgeon as the surgeon approaches the boundary. Further still, the OTTCAS system may be configured to provide feedback related to the controlthe operation of the OTT equipped surgical tool in response to theproximity of the tool to the resection boundary and the correspondingOTT CAS data processing response and resulting CAS outputs.

As described above, the OTT CAS system provides for the pre-operativeanalysis of a patient model and the identification of the tissue to beresected. After the portion of the tissue to be resected is determined,the OTT CAS system may analyze the data for the model and identify theboundary for the resection. The tissue to be resected may then beidentified in the OTT projector output using a plurality of colors basedon the relation to the resection boundary.

For instance, the OTT projector output may be adapted based on OTT CASprocessing factors to project onto a portion of the tissue that is notto be removed in red. Optionally, the OTT projector output may indicatea portion of the tissue that is to be resected that is relatively closeto the resection boundary in yellow. In still another alternative, theOTT CAS processes may produce an OTT projector output whereby theremainder of the tissue to be resected may be eliminated in green. Inthis way, as the surgeon views the surgical field during a procedure thesurgeon may cut rapidly and aggressively while the OTT projector outputindicates the tool is operating on tissue in the green zone. As thesurgeon approaches the resection boundary, the OTT-based projectoroutput indicates the tool is operating on tissue in the yellow zone.These OTT CAS determined projector outputs serve as indications to thesurgeon to proceed more slowly as the tool approaches the resectionboundary. In this way, the OTT CAS system provides a readilyidentifiable visual and graphical display directly onto the surgicalfield that informs the surgeon of the proximity of the current surgicalaction to a resection boundary. Similarly, the OTT CAS system can beused to visually recognize and use an OTT-based projector output toidentify the proximity of the surgical tool to sensitive anatomicalstructures, such as nerves, vessels, ligaments etc. OTT CAS output tothe projector may include distinctive color schemes to identify thestructures within the surgical field as part of OTT CAS output for theuser.

FIGS. 37A-44 relate to various alternative tactile feedback mechanismsalong with related kinematic responses and design criteria.

FIG. 37A illustrates a bent form that deflects to move an actuator inresponse to trigger force. FIG. 37B illustrates a sliding trapezoid formthat will deform and restore its shape in response to trigger force.FIG. 37C illustrates a rotating reader or encoder used to provide arotating response to the trigger force. FIG. 37D illustrates a framemoving in response to trigger force to depress a shaft into a base wherethe movement of the shaft may be registered as an indication of triggerforce. FIG. 37E illustrates a pinned element that may deflect toindicate an amount of trigger force.

FIGS. 38A and 38B illustrate a simple four bar mechanism, in a raisedand lowered, positions respectively that may be used to register triggerforce and displace a shaft.

FIGS. 39A, 39B 39C each illustrate a scissor mechanism 80 without aposition restoration element (39A) and driving an actuator 80, with atension spring as a position restoration element 84 (39B) and acompression spring as a position restoration element 84 (39C). Themovement of the actuator shown determines the height of the upper end ofthe scissor arms therefore the elevation of the scissor mechanism. Thisheight will press against, and will be felt by the user placing his orher finger on the tool trigger.

FIGS. 40A and 40B illustrate a side view of a scissor mechanism in araised and lowered configuration, respectively. The scissor mechanism 80includes a first link 86 and a second link 88 coupled at a pivot pointwhereby movement of the scissor raises and lowers the first and secondplatforms 90, 92. A position restoration element 84, here shown as aspring, is coupled to one end of the second link and to an actuator 82.The platforms have a length of about 22 mm and a maximum rise of about20 mm in the elevated condition shown in FIG. 40.

FIGS. 40C and 40D are charts relating to the displacementcharacteristics of the scissor mechanism 80 of FIGS. 40A and 40B. FIG.40C relates a platform trajectory with a height of the device. FIG. 40Drelates to the scissor angle with the displacement variation of thedevice.

FIG. 41 illustrates another scissor mechanism 80 having a surgeon systemoverride capability. The override capability is provided via theinclusion of a spring in line with the force application through theactuator. The actuator may be a component 140 is used for providing orreceiving OTT CAS data during computer assisted surgery procedures. Inthis aspect, the on tool tracking device includes a component 140adapted and configured to translate a movement received from a feedbackmechanism, such as from the shaft 80 relative movement into a signalused in a computer assisted surgery procedure. The component 140 may beprovided in a number of different configurations such as an encoder, anactuator or a motion transducer. In one aspect, the signal relates tothe operation of the surgical tool operated by the trigger. In still afurther embodiment, the component is or is adapted to include anactuator to impart movement to the shaft to influence the relativemovement between the first platform and the second platform. In afurther aspect, the actuator is configured to impart movement to theshaft in response to a signal related to controlling the operation ofthe surgical tool during a computer assisted surgery procedure.

The illustrated scissor mechanism embodiment shows the relationship ofthe first platform 90 and the second platform 92 borne by the links 86,88 of the scissor mechanism 80. In addition, this embodiment shows ascissor mechanism having a pair of position restoration elements used inconjunction with the scissor mechanism 80. One position restorationelement is the return spring positioned within the scissor mechanism 80.Another position restoration element is the override spring positionedbetween the scissor mechanism and the actuator or component 140.

FIG. 42 illustrates a scissor mechanism similar to the schematicmechanism illustrated in FIG. 41. The scissor mechanism 80 includes afirst platform 90 and the second platform 92 connected at one end of thelinks 80, and 86 in the pivoting relation to the first and secondplatform and sliding relation with the other end of the links 88, 86. Aposition restoration element, here a spring, is placed between theactuator or cable and a sliding and of a scissor link 88. Thisembodiment also includes the details of the elongate slots the first andof the platforms to permit sliding movement of the link first endrelative to the first and second platform. The second end of the links88, 86 are coupled in pivoting relation to the first platform and thesecond platform 90, 92. Here the motion of the first and secondplatforms is adjusted to the use of the spring or under the influence ofthe actuator. The operational characteristics of the mechanism of FIG.42 are better appreciated with reference to the charts and FIGS. 43 and44.

FIG. 45 is an isometric view of a tactile feedback mechanism. FIGS. 45and 46A illustrate isometric and side views of a tactile feedbackmechanism 150, respectively. The view of FIG. 45 shows the base plate152 use for attachment to a surgical tool 50 adjacent a trigger 52. Thescissor mechanism (best seen in FIG. 46A) is covered by a cover 191 thatis borne by the first platform 183 and moves along with the platform. Anactuation cable 82 is coupled to the scissor mechanism and moves inresponse to movement of the scissor mechanism.

FIG. 46B illustrates an isometric view of the scissor mechanism 155 ofFIG. 46A without the cover 191 or the platforms 183, 184. The Y-shapedlinkage 160 and 165 are pinned 163 to form a scissor mechanism 155. Aposition restoration element 84 is positioned between the first ends ofthe first link and the second link. Also visible in this view are is theshaft 173 used to slide along the slots 178 in the platforms.

FIGS. 46A-46F illustrate various views of the components and operationof the mechanism of FIG. 45. FIGS. 46C and 46D show the TFM 150 of FIGS.45 and 46A in an extended condition with (FIG. 46D) and without (FIG.46C) the top platform 183. The cable 82 is moved a displacement +y fromthe lower platform 184 in relation to the length of movement of thelinks along the slots 178.

FIGS. 46E and 46F show the TFM 150 of FIGS. 45 and 46A in an closed orretracted condition with (FIG. 46F) and without (FIG. 46E) the topplatform 183. The cable 82 is moved a displacement +x from the lowerplatform 184 in relation to the length of movement of the links alongthe slots 178.

FIGS. 47 and 48 are side views of an OTT 100 on a surgical tool 50having a TFM 150 positioned adjacent the trigger of the surgical tool.The actuator 82 extends from the TFM into the OTT 100. A component 140within the OTT is configured to receive and provide output to or receivefrom the TFM. In this embodiment, the cover 191 is expended away fromthe base 152 exposing a portion of the base 184.

When the TFM moves the cover 191 into the position show, the triggerfunction on the surgical tool is impaired by the cover 191 that blocksaccess to the trigger 152. FIG. 48 illustrates the cover 191 in alosered configuration where the trigger 52 is accessible.

FIGS. 47 and 48 illustrate a side view of an on tool tracking devicemounted on a surgical instrument having a tool (here a saw) with thetactile feedback mechanism of FIG. 45 in position to interact with thetrigger of the surgical instrument. FIG. 47 illustrates the tactilefeedback mechanism in an expanded configured that covers the trigger andFIG. 48 shows the tactile feedback mechanism collapsed to expose thetrigger.

FIGS. 49A-49B illustrate another alternative of a tactile feedbackmechanism in an open or expanded state (FIG. 49A) and a closed state(FIG. 49B). FIGS. 49C-49E illustrate the various views of the internalmechanisms of the devices in FIGS. 49A and 49B.

The FIGS. 49A and 49B illustrate isometric views of an over the triggertactile feedback mechanism 600 in a raised and lowered condition,respectively. The over trigger tactile feedback mechanism 600 has atrigger adapter 605 attached to the first platform 183. A modifiedtrigger seed text and is adapted to engage with the trigger 52. Themodified trigger seed fits within and is movable relative to the triggeradapter 605. A scissor mechanism 155 is provided as before to move thefirst platform and the second platform.

The relative positions of the platforms in views illustrate how in thecollapsed condition the modified trigger seat 610 is raised above thetrigger adapter 605. In contrast, in the raised condition the modifiedtrigger seat 610 is withdrawn within and below the upper surfaces of thetrigger adapter 605.

FIG. 49C is an isometric view of the scissor mechanism 155 in a raisedcondition with the upper platform and the trigger adapter removed. FIG.409D is similar to the view of FIG. 409C with the upper platform 183attached to the scissor mechanism 155. An aperture 620 is provided inthe upper platform 183. The aperture 620 used to provide couplingbetween the modified trigger seat 610 and the trigger 52.

FIG. 49E is similar to the other embodiments with the addition of thetrigger adapter 605 in position on top of the first platform 183. FIG.50 illustrates an embodiment of an OTT 100 coupled to a surgical tool 50where the trigger 52 of the tool 50 is covered by the tactile feedbackmechanism 600.

In the configuration of FIG. 50, a user's ability to manipulate thetrigger 52 is covered by the operation of the tactile feedback mechanism600.

FIG. 50 illustrates an embodiment of an OTT coupled for use with asurgical tool having an embodiment of the mechanism of FIGS. 49A and 49Bmounted for cooperation with the trigger of the surgical tool andconfigured to send and to receive trigger related with a component inthe OTT.

FIG. 51 is an alternative embodiment of a scissor mechanism utilizingtwo position restoration elements. FIG. 51 illustrates a scissormechanism similar to FIG. 42. In contrast to the scissor mechanism ofFIG. 42, the illustrated scissor mechanism in this embodiment includes apair of position restoration elements. One position restoration element84 is a return spring extended between the first and second platformsand coupled to the first ends of the links 86, 88. The return spring isused to modify the movement platforms and hence control triggerresponsiveness. The other position restoration element is the overridespring extending along the second platform. The override spring iscoupled to a sliding and of the link 88 and the cable 82. The returnspring in the override spring work in concert to provide a variety ofdifferent responsive features to the tactile feedback mechanism asschematically represented by FIG. 51. As a result the use of more thanone in different types of position restoration element provides a widevariety of response characteristics for the tactile feedback mechanismsdescribed herein.

FIGS. 52A and 52B illustrate front isometric and rear isometric views,respectively, of another OTT embodiment coupled to a surgical tool 50.OTT 700 includes a housing 710 having a camera mount 705 and projector710. In this embodiment, the camera mounts 705 is on the upper surfaceof the housing 710. The mount 705 contains a pair of cameras 707directed towards the tool 74 for imaging the active element 56. Inaddition, this embodiment includes a TFM hundred over the trigger of thetool 50. The cable 80 provides an interface between the TFM 600 and theOTT 700 for the various purposes of tactile feedback as describedherein. The OTT 700 also includes a display 702 on the upper surface ofthe housing 710. The display 702 may be used to provide OTT CAS outputinformation for the user. Additionally or alternatively, display 702 isused as a user interface for user inputs. The display 702 may beconfigured as a graphical user interface (GUI) or other type of computerinput device. Also shown is a computer in communication with the OTT 700for the purpose of utilizing the information obtained from the use ofthe OTT during a CAS procedure in furtherance of the completion of acomputer aided surgery. The computer includes within an electronicmemory accessible to the processing unit instructions for on tooltracking computer assisted surgery. In one embodiment, computer isincluded within the OTT 700 as part of the electronics package withinthe housing. In another embodiment, the computer is an externalcomponent configured for receiving and transmitting data related to OTTCAS processes either wirelessly or via a wired connection to and fromthe OTT 700.

As the above examples in the illustrative embodiments make clear,embodiments of the TFM mechanisms of the present invention may beadapted or configured to provide outputs related to trigger movement orposition or for further processing by the OTT CAS computer. The variousTFM mechanisms provided herein may be used to provide in a minimallyintrusive manner an indication of tool operation, characteristics orparameters (speed, position, rotation, setting, power level and thelike) for use by the OTT CAS system. An output from a tactile feedbackmechanism may be provided via an encoder/reader in the mechanism, in theOTT device, or mounted on the surgical tool itself. Still further,feedback mechanism embodiments may include wireless communications fortransmitting tactile feedback mechanism information or triggerinformation for further processing in the OTT device or the OTT CAScomputer. In a still further aspect, one or more components of thetactile feedback mechanism may be driven under instructions receivedbased on OTT CAS processes, modes or algorithms. In some embodiments,tactile feedback mechanism indications and data are used to provide adynamic real-time feedback loop from the OTT CAS system. Indicationsfrom the tactile feedback mechanism may also be used to provide theautomatic control of one or more surgical tool control features such as:the tools motor, actuator attenuating its motor/cutting/drilling actionspeed or stopping it as part of an appropriate OTT CAS processingoutput. In one aspect, the feedback loop control is provided based on adetermination of the OTT CAS system that automatic intervention ofsurgical tool functionality is needed to prevent an improper cut, orharm to an anatomical structure within the OTT CAS surgical field.

In still further aspects, embodiments of the tactile feedback mechanismor other feedback mechanisms configured to utilize the outputs from thesystems and methods described herein may be used to automatically orsemi-automatically control one or more operating characteristics of anactive element of a surgical tool utilizing an on tool tracking device.Still further an embodiment of the OTT CAS system may also be configuredto control the operation of the surgical tool in response to adetermination of the position of the surgical tool relative to thedesired boundary. Specifically, if the system determines that the toolis positioned within the tissue to be resected that is not proximate theboundary (i.e. in the green zone), the system may allow the surgicaltool to controlled as desired by the surgeon. If the system determinesthat the tool is positioned within the tissue to be resected that isproximate the boundary (i.e. the yellow zone), the system may reduce orattenuate the operation of the surgical tool. For instance, if the toolis a saw, and it enters the yellow zone, the system may slow down thereciprocation or revolution of the saw as it moves proximate theresection boundary. Further still, if the system detects that the toolis positioned at the boundary or on tissue that is not to be resected oroperated on, the system may control the surgical tool by completelystopping the tool. Although the system may automatically control theoperation of the surgical tool, the system includes an override functionthat allows the surgeon to override the control of the tool. In thisway, if the surgeon determines that a portion of tissue should beresected that was not identified for resection during the pre-operativeanalysis; the surgeon can override the system and resect the tissueduring the procedure.

Embodiments of the tactile feedback mechanism include a wide variety oftactile stimulus. For example, the stimulus could be as simple asenhanced vibration to indicate deviation of the surgical path from theintended resection. Tactile stimulus provides the opportunity for moresophisticated indications in accordance with the various modificationsand outputs provided by the OTT CAS methods described herein.

In general, powered surgical tools are activated by means of a triggerand embodiments of the feedback based mechanisms described hereinprovide detectable and variable (increases and decreases under controlof the OTT CAS computer) resistance on the trigger or pressure on thesurgeon's finger actuating the tool in a manner to indicate to thesurgeon when the surgical path or current use of the active elementdeviates from the intended resection or other action according to theOTT CAS surgical plan. It is to be appreciated that the variety ofdifferent configurations for providing tactile feedback may be used withan unmodified, modified or replaced trigger for actuating the surgicaltool used with an OTT device. In some various alternative embodiments, atrigger based feedback assembly includes a dynamic member coupled to ascissor mechanism that is in turn coupled to a stationary base (usuallymounted on the handle of the surgical tool. The position or stiffness ofthe assembly, typically as a result of interaction with a transmissionshaft or cable is dictated by a control unit within the OTT. The controlunit may be configured to provide a wide variety of OTT related feedbackfunctions including, by way of example, an actuator to operate thetransmission shaft which in turn changes the force to close the scissormechanism, moves the trigger mechanism to a full extended position, movethe trigger mechanism to a full contracted position, move to a positionto impair operation of the trigger, or, optionally to stop operation ofthe active element of the tool. In one aspect, the transmission shaft orcable or element is Bowden cable. In still other embodiments, thetransmission shaft that couples the scissor mechanism to the associatedcomponent in the OTT may be any suitable element such as a rod, spring,solenoid, chain, gear, or a mini pneumatic or hydraulic actuated system.Still further, it is to be appreciated that the actuator used for thecontrols described above may also be included within the feedbackmechanism in proximity to the trigger. In one alternative of thisaspect, the actuator may be connected to the OTT device via a wired orwireless connection to provide the appropriate OTT CAS process controlsignals to the actuator in furtherance of the above described OTT CAStechniques.

The control unit is also capable of receiving data from the computersystem. When the system determines a deviation in excess of a specifiedthreshold level exists between the surgical path and the surgical planby comparing the position of the tool to the intended resection of thesurgical plan, the control unit actuates the transmission, increasingthe resistance required to pull the trigger. Indication can be providedin the form of preventing the depression of the trigger so that thesurgeon cannot activate the tool. Alternatively, indication can take theform of increased resistance, which the surgeon can overcome by theapplication of more force.

The trigger and other tool control embodiments described with regard toFIGS. 37A-51 may also be utilized with an externally tracked tool suchas those described in co-pending and commonly assigned application Ser.No. 11/764,505 filed on Jun. 18, 2007 and Ser. No. 11/927,429 filed onOct. 29, 2007, each of these applications are incorporated herein byreference in its entirety.

FIGS. 52A and 52B are front and rear isometric views respectively of anon tool tracking and navigation device (OTT) that includes a displaywith OTT housing coupled to a surgical tool having a trigger basedfeedback mechanism coupled to the OTT. The view also shows an exemplarycomputer system in communication with the OTT.

FIG. 36 is a flowchart representing an exemplary OTT CAS processincluding modification of any of the above described OTT CAS processesto include associated surgical tool operational characteristics,parameters or other data related to the use of an active element in anyOTT CAS process or procedure. The OTT CAS process 3600 includes many ofthe same processing steps described above with regard to OTT CAS process3100 in FIG. 31A.

FIG. 63 illustrates a flowchart 6300 illustrating the various stepsperformed by the CAS guidance system when operating in hover mode. Thesteps start by getting bones and tool positions in registration at step6302. Next, at step 6304 calculate deviations (i.e. bones and toolserrors relative to the plan). Next at step 6306, determine whether thedeviations calculated are less than or equal to TH−1. TH−1 is the outerthreshold spacing. In this context, the outer threshold spacing is usedto determine when the tool is at a distance spaced sufficiently far awayfrom the point of surgery that certain aspects or secondary operationsmay be utilized with system resources, or that high tolerance trackingor control is not critical. If the answer at step 6306 is yes then theprocess proceeds to step 6308. At step 6308, the calculated errors arecompared against a smaller deviation which is threshold TH−2. Thethreshold TH−2 is used as an inner threshold value to trigger when thesystem is close in to the surgical field. If the answer to step 6308 isyes then the method proceeds to step 6310 to determine whether this isthe first time that the threshold TH−2 has been triggered. If the answerat 6310 is yes then the method proceeds to step 6312 where all secondarytasks are not permitted to operate, and FIG. 63 shows examples of thosein box 6312. In step 6312 the system is essentially overriding all otheroperations so that maximum resources are made available for the trackingmode since the comparisons at step 6306 and step 6308 have determinedthat the system is near or is within cutting mode. Examples of secondarytasks that would not operate during this time include, for example, RFrecalibration, data backup, proximity to registration and various datatests performed by the system. After step 6312, the next step 6314 setsthe bone as the center of the screen on the display used by the OTT,unless overridden by the user or the user has preferences set to thecontrary. Next at step 6316, additional control signals are sent withinthe system. In the illustrative steps of 6316, motor control is turnedon, 2-D guidance is turned on, projector is turned on in the OTTembodiment. Again, here and from this point onwards, the descriptionassumes that the user has not set options to the contrary of what isdescribed here. If an ETT system is being used, the iPod screen is alsoturned on and a suitable user selectable default initial view shown. Inaddition, the navigation and error calculations functions remain inoperation. Next at step 6318, various slew rates are set to 100%. In theillustrative step of 6318, navigation, error calculations, motor controland communications, 2-D guidance, projector, and iPod screen are all setto 100%. Next at step 6320, this mode of operational loop is repeatedand the system continues to get bones and tool positions at step 6302.

Continuing on from 6302 to calculate bones and tools errors at 6304,next at step 6306, if the response at step 6306 is “no” then the systemproceeds to step 6322 to determine whether or not this is the first timethat the system has registered an error that is greater than the nearthreshold TH−1. If the answer is yes to step 6322 the method proceeds tostep 6324 which permits some aspects of the system to be placed intodifferent states. Next at step 6326, the slew rates are set to a varietyof different levels in contrast to the slew rate settings found in step6316. Next at step 6328, secondary tasks may be performed by the system.At step 6328, secondary tasks are allowed and system resources may bedevoted to other activities since the system is likely not in cuttingmode. Thereafter, the system returns to the base step of 6302 to getbone and tool position information. Returning down the method from 6302to the calculation steps 6304 and the smaller deviation comparison fornear threshold TH−1, if the answer at step 6306 is yes and the answer atthe near field deviation TH−2 (step 6308) is no, the method thenproceeds to decision step 6330. If the answer to the question first timeat 6330 is no, indicating that this is not the first time that the nearthreshold error has been greater than the error threshold TH−2 then themethod returns back to step 6302 to get bone and tool information. If,the answer to first time query at step 6330 is “yes”, then the systemproceeds to step 6332. In step 6332, various control functions are setto different values based upon the computer's determination of the toolposition. Next at step 6334, various slew rates are set for navigation,error calculations and 2-D guidance. Thereafter, at step 6336 secondarytasks are also allowed to operate similar to step 6328. The secondarytasks are permitted because the system has determined that systemresources may be used for other than critical navigation with motorcontrol functions simultaneously. In each of the first time blocks, 6322and 6330 and 6310, this is a simplification for a validation andlatching process to prevent repeated switching of states when notnecessary and adding some hysteresis to prevent toggling back and forthfrom one state to another based on random fulfillment of a condition. Bysetting the thresholds TH−1 and TH−2 to appropriate levels then thesystem may determine whether or not a user's movement of the OTT isintentional and directed away from the field of surgery or intentionaltowards the field of surgery or continuing on a step of cutting withonly a minor adjustment, for example. Such intended hysteresis of coursereduces the effect of digital noise and random errors especially ner theboundaries of different states of the system.

In general, in the method 6300, the left hand steps (6328, 6326, and6324) indicate a normal hover mode where the system liberates resourcesfor secondary tasks when time sensitive tasks are not required. On theright hand side of the method 6300 (steps 6332, 6334 and 6336) are usedwhen the system indicates that it is within a volume of interestrelative to the target bone but still not in a position to cut thetarget bone (like a standby when the sensors and resources would beavailable to switch motor control on at short notice). Secondary tasksare still allowed in this condition, but time sensitive aspects are moreclosely monitored than in the previous case described above on the lefthand side. In the bottom portion of the method 6300, these indicate thetime-sensitive tasks are in action during active cutting. Method steps6312, 6314, 6316, and 6318 are all used to insure that full slew ratesare applied to all cut-related processes. During this time, systemresources are not directed towards secondary resources or secondaryactivities are neglected all together.

In general, in the method 6300, the left hand steps (6328, 6326, and6324) indicate a normal hover mode where the system primarily saveselectric battery power and reduce heat generation and dissipation andliberates resources for secondary tasks when time sensitive tasks arenot required. On the right hand side of the method 6300 (steps 6332,6334 and 6336) are used when the system indicates that it is within avolume of interest relative to the target bone but still not in aposition to cut the target bone (like a standby when the sensors andresources would be available to switch motor control on at shortnotice). In still another aspect, an additional factor or considerationin steps 6326, 6324, 6332, or 6334 is that one or more electronicdevices may be shut down, placed in standby mode or otherwise adjustedto save power. As a result of this type of determination by the OTT CASsystem, it is believed that battery life in an OTT module may beextended because high energy consuming devices like the projector, forexample, may be placed in an energy conservation mode if the OTT CASmode deems that a practical step.

FIG. 64 illustrates a simplified hover mode state diagram. The modestate diagram begins at initiation at step 6405. Next, the system mayenter into hover mode at step 6410. Thereafter, if system parametersindicate that bone registration is being performed the system will moveinto a bone registration mode at step 6415. At the completion of boneregistration, the system may either end tracking or return to theinitiation step 6405. Alternatively, at the conclusion of boneregistration the system may set hover mode and return to the hover modestep 6410. In addition, from the hover mode step 6410, the system maydetect bone cutting steps. In this case, the system will go into bonecutting mode as shown at step 6420. At the conclusion of the bonecutting step, the system may return to hover mode at step 6410, or ceasetracking and return to initial mode 6405. Another option from hover mode6410 is to move into a bone implant fit assessment at step 6425. At theconclusion of any implant fit assessment, the system may return to hovermode at 6410, or cease tracking and return to initial mode state 6405.One example of assessment (that is not shown in the diagram to avoidclutter) si to assess the quality of the cut with a navigated surfacetester, with which a cut surface location and orientation are tested toassess their quality and suggest further cutting refinements if needed.Still another alternative path from hover mode 6410, is to go intomid-range tracking at step 6430. From the mid-range tracking step 6430,the system may cease tracking and return to the initial state 6405.Alternatively, the mid-range tracking step 6430 may conclude and returnto the hover mode tracking step 6410.

FIG. 65 illustrates another alternative view of the hover modeoperation. In the sequence illustrated by FIG. 65, the system is shownmoving between three modes; hover mode 6505, a bone cutting mode 6510 orin implant or cut fit assessment mode 6515. When in the hover mode 6505,using a saw and with the bone being close to an instrument will move thesystem into a bone cutting tracking mode 6510. Alternatively, as thebone is moved further away from the instrument, or vice versa, the sawaway from the bone, the system will detect such movement and move fromthe bone cutting mode 6510 and back into a remote 6505 hover mode.Alternatively, if the system detects a navigated implant trial ornavigated bone (cut) surface assessment tool are visible or implanttrial or such tool is close to the bone, then the system will shift fromhover mode 6505 into the implant fit or bone surface assessment step6515. At the conclusion of the assessment above, as when the bone is nowfar from the trial implant or assessment tool, the system will return tohover mode 6505. FIGS. 66A, 66B and 67 illustrate various in-roomdisplay and on tool display or projector views depending on theoperation of the OTT. Turning now to FIG. 66A, the in-room scene (A)illustrates an active cutting step. Since an active cutting step isinvolved, the on tool display (portion B of the view of FIG. 66A) isindicating the angular error (orientational deviation around two axes)or other cutting information about error rate or location of the bladerelative to a surgical plan (offset) as described herein. In the view ofFIG. 66B, the in-room display is showing the side view of the tool andblade in contact with the bone according to the surgical plan.

Determining orientation for the 2D guidance display

In various places of our graphical user interface (GUI) on the main CAScomputer, we sometimes use our flight simulator like (2D) graphicalguidance system 66B. This display guides the user to move the instrumentso the plane (labeled) merges with the target surface plane (labeled) bytipping, or changing the pitch of, the saw downwards, and rolling, tomake the two lines coincide with each other (hence saw pitch is correct)AND both lie along the horizon line (hence saw roll is correct). Whetherthe guidance lines should go up or down depends on whether the navigatedsaw was held normally or upside down—and latter is possible.

To determine if the guidance to go up or down, depending on whether thesaw is upside down or normal, is for the computer which logs positionsto store recent history (eg. a few ms or seconds) and examine the movingaverage. If upon reviewing the last one hundred or ten or say one secondof tracking, the computer notes that it is telling the user to go up andyet we are going down, then it must be that we are holding the sawupside down. So if it finds that the user is getting further away whilewe they are trying to move towards the target, it switches the guidanceby 180 degrees and tells you so verbally (by voice). If you want tooppose that function and override it, you can optionally stop that.

Also, the computer can tell if you are almost aligned (near the targetin 3D) and within a few degrees. Then it knows that you are in the rightorientation. But if you are almost 180 degree upside down to the target(i.e. parallel to the target but within almost 180 degrees) then itmeans that you are holding the saw upside down so it automaticallyswitches its coordinate system to adjust. If it sees you persistentlyholding the saw at 180 degrees towards the target plane, (you are closeto the target plane but you are holding it at about 180 degrees plus orminus a certain threshold, say plus or minus say 10 degrees) then itautomatically switches the guidance to be the other way round so theguidance is effectively going in the correct direction.

The concept relies on a knowledge based system and the followingproviso: The user almost knows what they are doing and they are almostright, but the system suffered a reversal of coordinate system sign dueto the user flipping the device upside down. We can make this detect andcorrect automatically within a few milliseconds or much less than asecond.

FIG. 67 shows the location of an OTT system relative to a bone on anapproach or an evaluation step. In the view of FIG. 67A, the in-roomview, the tool is shown approaching the surgical field. The in-roomdisplay B also shows the approach of the tool to the bone within asurgical field. The view of FIG. 67C shows the display on the on toolsystem which indicates the alignment of the tool relative to the bone.The view shown in FIG. 67C is adjustable using the smart views commandas described herein and elsewhere.

In addition or alternatively, any of the OTT modules described hereinmay be modified to have additional functionality. For example, an OTTmay be modified to include a display. Alternatively, the OTT may beadapted to work with a remote control to drive the main system, such as,for example, to run via an iPod, an iPad or other iOS or Android (orsmart phone like) device that may be removeably mounted on the OTT. Inother aspects, such an OTT may be described as an OTT tablet. In oneembodiment, an OTT module may have a screen (eg. color LCD type) orother display incorporated into a surface of the OTT housing. In analternative embodiment, a display is provided as a detachable item. Inone embodiment, the display runs on an iOS implementation and runs oniPod, iPads, etc. In addition or alternatively, an iPod or other devicecan be used as a ‘remote control’ to drive the main system. That is, theremote control device can be on-board the OTT device, or just loose. Inuse, an iPod, iPad or smart phone like device for this purpose is placedin a sterile bag and put it in the surgical scene, so the surgeon and/ornurse can drive the system settings from there.

Portable Display Screen

The attached screen is currently embodied as an iPhone and could be anyother similarly-sized smart phone, such as a Droid or Blackberry, or acustom built touch display.

Attached to the saw, the display is typically intended for use with anattitude and offset distance display. It can also utilize a 3D renderingengine software and show 3D surface or volumetric models and provide thesame guidance and selection of viewing parameters as specified in theautomatic selection of view.

Additionally, the user can move the model on the screen. Such changesare analogous to the view on the main OTT CAS screen with the advantagebeing the closer proximity of the attached screen compared to theterminal screen, and the implications of touching screens in the sterileenvironment versus main computer screens which may (optionally) or maynot be sterile or conveniently close to the surgeon or assistant.

In another example, the view, or any parameters of the display, can bechanged by using the touch screen interface.

The attached screen can also be removed and used as a detached displayor as a remote control device.

In still another aspect, there is provided methods of using thepico-projector or other projector onboard the OTT for use in anautomatic, or semi-automatic bone registration technique. In one aspect,there is provided a method for calculating or determining the boneregistration matrix in the context of OTT using reference frames. Thiscan be implemented as a combination of the 3D tracking described for OTTand a dynamic 3D scanning process such as those used in commerciallyavailable image processing and tracking processes.

In one aspect, such an OTT based registration process or techniqueincludes the steps of:

a) Obtaining a 3D model of the anatomy (e.g. bone), usually duringpre-surgical planning. For example, on an image-based setup, this can bedone as 3D reconstruction from the patient's computer tomography (CT) orMagnetic resonance Imaging (MRI), data or through morphing (scaling) ofa generalized bone from an atlas.b) Attaching a tracking reference frame to the bone. The trackingreference frame is visible to the OTT cameras.c) Performing a 3D scanning of the anatomy (e.g. bone) surface by usingOTT's projector to project a pattern (e.g. point(s), line(s), grid(s),etc.) on the surface of interest and OTT's camera(s) system to captureand process the reflection of the lights on the surface of interest.d) Simultaneously with c), the tracking in 3D the reference frameattached to the object of interest (e.g. bone), using any of thetechniques described herein. While OTT cameras are used for bothprocesses, 3D scanning and tracking, one example of how to coordinatethe two processes is by switching from one function to another at highrate, and pairing each 3D scanning data sampling with a 3D trackingposition/orientation.e) Based on data from c) and d), obtaining a surface model of theanatomy (e.g. bone) surface positioned and oriented relative to thereference frame attached to the object of interest (e.g. bone).f) Surface matching a) and c). This process calculates a transformationmatrix that matches one surface into the other. The process can be donemanually (with user graphical intervention or verification) or withvarious levels of automation. The latter harnesses image processing andpattern recognition and matching routines using correlation or otherknown techniques.g) Calculating final anatomy (e.g. bone) registration matrix combininge) and f).

The process described above may be modified or enhanced using a numberof different variations. Some variations of the steps outlined aboveinclude, by way of illustration and not limitation: (a) usingpico-projector for bone registration is similar to the steps above butoptionally includes using different wavelengths filters to optimize stepd); or (b) using pico-projector for bone registration is similar to thesteps above but optionally includes using the known anatomy shape froma) to optimize 3D scanning process on c).

OTT Tracking without Reference Frames

In this alternative embodiment, an OTT system is adapted and configuredfor performing reference frame-free 3D tracking with OTT. In one aspect,there is the step of projecting a known pattern with the projector (e.g.mono- or multi-chrome, infrared, etc. point(s), line(s), grid(s), etc.)on a known geometry (e.g. bone), and applying image recognition andcomputer vision algorithms on the reflected light to track the positionand orientation of the object of interest (e.g. bone) in 3D (e.g.relative to OTT's internal origin and coordinate system). This may beconsidered a form of using a projected grid for navigation. One methodfor implementing such a freehand surgical navigation technique includes,by way of example and not limitation:

a) Obtaining a 3D model of the anatomy (e.g. bone), usually duringpre-surgical planning. For example, on an image-based setup, this can bedone as 3D reconstruction from the patient's computer tomography (CT)data or other methods mentioned above.b) Dynamically projecting a known pattern with the projector (e.g. mono-or multi-chrome, infrared, etc. point(s), line(s), grid(s), etc.) on thereal patient's anatomy (e.g. bone).c) Applying image recognition and computer vision algorithms (as well astechniques presented in 2) on the images projected on the anatomy (e.g.bone) to calculate its position and orientation in space.

The process described above may be modified or enhanced using a numberof different variations. Some variations of the steps outlined aboveinclude, by way of illustration and not limitation: (a) using OTT'sprojector for both, 3D tracking and displaying information to guide theuser during cutting, drilling, etc., the system uses different colorschemes to two sets of images to avoid interfering with the imageprocessing, as well as interfering with the users' interpretation of theprojected guidance; (b) using emitted infrared light for trackingpatterns to avoid interfering with the users' interpretation of thevisible light projected guidance; (c) using OTT's switches from grid toguidance at high rate to create a stroboscopic effect, but stillpreventing the two processes (object tracking and user guidance) frominterfering with each other.

Multiple Reference Frames

For a particular surgical case there may not be a single location forthe bone's reference frame where the instrument with the cameras can‘see’ it from any location required for cutting (or drilling, or filing,etc.). In such cases, one can use a ‘combination’ reference frame(multi-faced): A single registration process (using any of the faces)allows the system to track the object afterwards regardless of which ofthe faces is visible at the time.

Notwithstanding, any element of the indicator subsystem could readily beused for any approach to computer assisted surgery wherein the computerassisted surgery system establishes both the location of the tool inthree dimensions and calculates where, according to a surgical plan, thesurgeon intends to make a resection. In one alternative aspect, themethods, systems and procedures described herein are modified toincorporate one or more of the techniques, devices or methods describedin U.S. Non Provisional patent application Ser. No. 11/764,505 filed onJun. 18, 2007 and published as US 2008/0009697 entitled “Method andApparatus for Computer Aided Surgery,” the entirety of which isincorporated herein for all purposes.

It will be recognized by those skilled in the art that changes ormodifications may be made to the above-described embodiments withoutdeparting from the broad inventive concepts of the invention. It shouldtherefore be understood that this invention is not limited to theparticular embodiments described herein, but is intended to include allchanges and modifications that are within the scope and spirit of theinvention as set forth in the claims.

1.-157. (canceled)
 158. An on tool tracking and guidance device,comprising: a housing having a surface for releasable engagement with aportion of a surgical tool; a first camera and a second camera in anarrangement where each of the first camera and the second cameraprovides an image output selected for viewing substantially all of asurgical field selected for a computer assisted surgery procedure, thefirst camera and second camera coupled to or within the housing; asensor coupled to or within the housing; a projector coupled to orwithin the housing configured to provide an output at least partiallywithin the surgical field of view; and an electronic image processorwithin or in communication with the housing configured to receive anoutput from each of the first and second cameras and perform an imageprocessing operation using at least a portion of the output from each ofthe first and second cameras for use in the computer assisted surgeryprocedure.
 159. The on tool tracking and guidance device of claim 158,wherein the sensor is selected from the group consisting of: aninclinometer, a gyroscope, a two axis gyroscope, a three axis gyroscopeor other multiple axis gyroscope, a one-two-three or multiple axisaccelerometer, a potentiometer, and a MEMS instrument configured toprovide one or more of roll, pitch, yaw, orientation, or vibrationinformation related to the on tool tracking device.
 160. The on tooltracking and guidance device of claim 158, wherein the surgical fieldincludes an active element of the surgical tool, wherein the activeelement is a saw blade, burr, or drill.
 161. The on tool tracking andguidance device of claim 158, wherein the output from the projector isadapted for projection on a portion of the patient's anatomy or on orwithin the surgical field.
 162. The on tool tracking and guidance deviceof claim 161, wherein the portion of the anatomy is a bone.
 163. The ontool tracking and guidance device of claim 161, wherein the adaptedoutput is adjusted for the curvature, roughness, or condition of theanatomy.
 164. The on tool tracking and guidance device of claim 158,wherein the electronic image processor is in wireless communication withthe housing and is located outside of the housing.
 165. The on tooltracking and guidance device of claim 158, wherein the computer assistedsurgery procedure is a freehand navigated computer assisted surgeryprocedure.
 166. The on tool tracking and guidance device of claim 158,wherein the surface for releasable engagement with a portion of the handheld surgical tool includes a saddle that is shaped to form acomplementary curve with the portion of the hand held surgical tool.167. The on tool tracking and guidance device of claim 158, furthercomprising: a display on the housing.
 168. The on tool tracking andguidance device of claim 158, wherein the projector is a pico projector.169. The on tool tracking and guidance device of claim 158, furthercomprising: a communication element within the housing configured toprovide information related to the image output to a component separatefrom the housing.
 170. The on tool tracking and guidance device of claim169, wherein the communication element provides information wirelesslyto and from the component separate from the housing.
 171. The on tooltracking and guidance device of claim 169, wherein the communicationelement provides information via a wired connection to the componentseparate from the housing.
 172. The on tool tracking and guidance deviceof claim 158, further comprising a third camera and a fourth camerawithin or coupled to the housing.
 173. The on tool tracking and guidancedevice of claim 172, further comprising a fifth camera and a sixthcamera within or coupled to the housing.
 174. An on tool tracking andguidance device, comprising: a housing having a surface for engagementwith a surface on a saddle, the saddle configured to engage with a handheld surgical tool; a first pair of cameras including a first camera anda second camera within or coupled to the housing; and a second pair ofcameras including a third camera and a fourth camera within or coupledto the housing, wherein when the housing is coupled to the saddle thefirst, second, third, and fourth cameras are in position to provide animage output having a field of view including at least a portion of anactive element of the hand held surgical tool coupled to the saddle.175. The on tool tracking and guidance device of claim 174, wherein thefirst and second cameras are near field stereoscopic cameras and thethird and fourth cameras are wide field cameras.
 176. The on tooltracking and guidance device of claim 174, further comprising a thirdpair of cameras including a fifth camera and a sixth camera within orcoupled to the housing.
 177. The on tool tracking and guidance device ofclaim 176, further comprising a fourth pair of cameras including aseventh camera and an eighth camera within or coupled to the housing.178. The on tool tracking and guidance device of claim 174, furthercomprising one or more sensors within or coupled to the housing. 179.The on tool tracking and guidance device of claim 178, wherein the oneor more sensors are selected from the group consisting of: aninclinometer, a gyroscope, a two axis gyroscope, a three axis gyroscopeor other multiple axis gyroscope, a one-two-three or multiple axisaccelerometer, a potentiometer, and a MEMS instrument configured toprovide one or more of roll, pitch, yaw, orientation, or vibrationinformation related to the on tool tracking device.
 180. The on tooltracking and guidance device of claim 174, further comprising: anelectronic image processor within or in communication with the housingconfigured to receive an output from the first and second pairs ofcameras and perform an image processing operation using at least aportion of the output from the first and second pairs of cameras infurtherance of at least one step of a computer assisted surgeryprocedure.
 181. The on tool tracking and guidance device of claim 174,wherein the first pair of cameras or the second pair of cameras comprisea physical or electronic filter for viewing within the infraredspectrum.
 182. The on tool tracking and guidance device of claim 174,wherein the first, second, third, and fourth cameras have a field ofview of from about 50 mm to about 250 mm.
 183. The on tool tracking andguidance device of claim 174, wherein a visual axis of the first cameraand a visual axis of the second camera are inclined towards one anotherrelative to lines generally parallel to a longitudinal axis of thehousing or of a surgical tool attached to the housing, wherein a visualaxis of the third camera and a visual axis of the fourth camera areinclined towards one another relative to lines generally parallel to thelongitudinal axis of the housing or of the surgical tool attached to thehousing.
 184. The on tool tracking and guidance device of claim 174,wherein a visual axis of the first camera and a visual axis of thesecond camera are inclined at an angle of between about 0° to about 20°relative to a line generally parallel to a longitudinal axis of theactive element of the hand held surgical tool coupled to the housing,wherein a visual axis of the third camera and a visual axis of thefourth camera are inclined at an angle of between about 0° to about 20°relative to the line generally parallel to the longitudinal axis of theactive element of the hand held surgical tool coupled to the housing.185. The on tool tracking and guidance device of claim 174, furthercomprising a projector coupled to or within the housing configured toprovide an output at least partially within the field of view.
 186. Theon tool tracking and guidance device of claim 185, wherein when thehousing is coupled to the saddle and the saddle is coupled to thesurgical tool the first, second, third, and fourth cameras and projectorhave a fixed spatial relationship to the saddle and the surgical tool.187. The on tool tracking and guidance device of claim 185, wherein anoffset distance between the active element of the surgical tool and theprojector, first camera, second camera, third camera, and fourth camerais determined based on the configuration of the saddle and engagement ofthe saddle with the surgical tool.
 188. A method for computer assistedsurgery (CAS) using a freehand surgical tool, the method comprising:creating a three dimensional representation of a portion of a patient towhich a bone or tissue cutting procedure is to be performed; identifyingan area of the three dimensional representation corresponding to theportion of bone or tissue for which the procedure is to be performedusing an active element of the freehand surgical tool; creating asurgical plan for the area of the three dimensional representationcorresponding to the portion of bone or tissue; determining a positionof the portion of bone or tissue for which the procedure is to beperformed; determining a position of the freehand surgical tool;calculating a distance between the position of the portion of bone ortissue and the position of the hand held surgical tool; setting a modeof the hand held surgical tool to a normal tracking mode if the distancebetween the portion of bone or tissue and the hand held surgical tool isgreater than a first threshold distance; setting the mode of the handheld surgical tool to an enhanced tracking mode if the distance betweenthe portion of bone or tissue and the hand held surgical tool is lessthan the first threshold distance and is greater than a second thresholddistance; and setting the mode of the hand held surgical tool to acutting mode if the distance between the portion of bone or tissue andthe hand held surgical tool is less than the second threshold distance.189. The method of claim 188, further comprising contacting the bone ortissue with the active element of the freehand surgical tool while thesurgical tool is in cutting mode.
 190. The method of claim 189, whereincontacting the bone or tissue with the active element includes making aplurality of planar cuts to a femur or tibia or knee.
 191. The method ofclaim 190, wherein the plurality of planar cuts are part of a total kneereplacement procedure.
 192. The method of claim 191, wherein theplurality of planar cuts are pre-selected based on a configuration of apre-determined prosthesis to be implanted in a patient.
 193. The methodof claim 192, further comprising changing the mode of the hand heldsurgical tool after performing the plurality of cuts to an implant fitevaluation mode.
 194. The method of claim 193, further comprisingcomparing the plurality of planar cuts to the surgical plan and theprosthesis to be implanted to determine the compatibility of the implantwith the plurality of cuts.
 195. The method of claim 188, furthercomprising repeating determining the position of the portion of bone ortissue and determining the position of the hand held surgical tool. 196.The method of claim 188, wherein the normal tracking mode and enhancedtracking mode allow for secondary tasks selected from the groupconsisting of: calculation of motion between a femur and tibia,recalibration of a reference frame, and determination of the hand heldsurgical tool proximity to a registration deck, wherein the cutting modedoes not allow secondary tasks selected from the group consisting of:calculation of motion between a femur and tibia, recalibration of areference frame, and determination of the hand held surgical toolproximity to a registration deck, wherein the cutting mode does notallow the secondary tasks.
 197. The method of claim 188, wherein settingthe mode to the normal tracking mode and the enhanced tracking modeincludes turning off a motor control function of the hand held surgicaltool, wherein setting the mode to the cutting mode includes enabling themotor control function of the hand held surgical tool.
 198. The methodof claim 188, wherein setting the mode to the normal tracking modeincludes turning off a two-dimensional guidance graphical interface(GUI) associated with the hand held surgical tool, wherein setting themode to the enhanced tracking mode and cutting mode includes turning onthe two-dimensional guidance GUI associated with the hand held surgicaltool.
 199. The method of claim 188, wherein setting the mode to thenormal tracking mode and enhanced tracking mode includes turning off aprojector on the hand held surgical tool, wherein setting the mode tothe cutting mode includes turning on the projector.
 200. The method ofclaim 188, wherein setting the mode to the normal tracking mode includesturning off a display on the hand held surgical tool, wherein settingthe mode to the enhanced tracking mode and cutting mode includes turningon the display.
 201. The method of claim 188, wherein changing the modefrom the normal tracking mode to the enhanced tracking mode includesincreasing resources appropriated to the navigation and errorcalculation of the hand held surgical tool.
 202. The method of claim188, wherein changing the mode from the enhanced tracking mode to thecutting mode includes increasing resources appropriated to thenavigation and error calculation, a tool motor controller, atwo-dimensional guidance graphical interface associated with the handheld surgical tool, and a projector or display on the hand held surgicaltool.
 203. The method of claim 188, wherein the first threshold distanceis greater than 200 mm and the second threshold distance is 100 mm to200 mm.
 204. The method of claim 188, wherein the second thresholddistance is 70 mm to 100 mm.
 205. The method of claim 188, wherein thesecond threshold distance is 10 mm to 0 mm.
 206. The method of claim188, further comprising setting the first threshold distance and thesecond threshold distance prior to determining the position of theportion of bone or tissue for which the procedure is to be performed.207. The method of claim 188, further comprising attaching a referenceframe including one or more position markers to the patient at apredetermined spatial orientation to the portion of bone or tissue,wherein determining the position of the portion of bone or tissueincludes determining the position of the reference frame.
 208. Themethod of claim 207, further comprising using a plurality of cameras todetermine the position of the one or more position markers.
 209. Themethod of claim 208, wherein the plurality of cameras are within orcoupled to the housing.
 210. A method for performing a computer assistedsurgery (CAS) procedure using a hand held surgical instrument having anon tool tracking device attached thereto, the method comprising:collecting and processing CAS data using the on tool tracking deviceincluding a position of the tool determined using data from a first pairof cameras and a second pair of cameras on or within a housing of the ontool tracking device; assessing the CAS data in real time during the CASprocedure; performing CAS related operations using the on tool trackingdevice by providing to a user guidance related to a CAS step; andproviding the user of the surgical instrument an output related to theassessing step by projecting or displaying the output related to the CASprocedure.
 211. The method of claim 210, assessing further comprising: acomparison of data received from the on tool tracking device and dataprovided using a computer assisted surgery surgical plan.
 212. Themethod of claim 210, further comprising: determining a predefinedcomputer aided surgery processing mode based on the results of theassessing step, the predefined processing mode selected from the groupconsisting of a hover mode, site approach mode, and active step mode.213. The method of claim 210, wherein the step of providing a CAS outputto the user is changed and an OTT CAS processing technique or output ismodified as a result of the user performing one or more steps of acomputer assisted surgery procedure on a knee comprising: making adistal femur cut, making a distal femur anterior cut, making a distalfemur posterior lateral condyle cut, making a distal femur posteriormedial condyle cut, making a distal femur anterior chamfer cut, making adistal femur posterior lateral condyle chamfer cut, making a distalfemur posterior medial condyle chamfer cut, making the distal femur boxcuts, drilling the cavity of a distal femur stabilization post, making aproximal tibial cut, making proximal tibia keel cut, or drillingproximal tibia keel's holes.
 214. The method of claim 210, wherein thecollecting and processing CAS data using the on tool tracking devicefurther includes receiving data from one or more sensors on or within ahousing of the on tool tracking device.
 215. The method of claim 214,wherein the one or more sensors are selected from the group consistingof: an inclinometer, a gyroscope, a two axis gyroscope, a three axisgyroscope or other multiple axis gyroscope, a one-two-three or multipleaxis accelerometer, a potentiometer, and a MEMS instrument configured toprovide one or more of roll, pitch, yaw, orientation, or vibrationinformation related to the on tool tracking device.
 216. The method ofclaim 210, wherein the position of the tool is calculated based on aspatial relationship between the position of the first and second pairsof cameras to one or more reference frames attached to a patient.
 217. Amethod for performing a computer assisted surgery (CAS) procedure usinga hand held surgical instrument having an on tool tracking deviceattached thereto, the method comprising: collecting and processing CASdata using the on tool tracking device including a position of the tooldetermined using data from a first pair of cameras and one or moresensors, the first pair of cameras and one or more sensors on or withina housing of the on tool tracking device; assessing the CAS data in realtime during the CAS procedure; performing CAS related operations usingthe on tool tracking device by providing to a user guidance related to aCAS step; and providing the user of the surgical instrument an outputrelated to the assessing step by projecting or displaying the outputrelated to the CAS procedure.
 218. The method of claim 217, wherein theone or more sensors are selected from the group consisting of: aninclinometer, a gyroscope, a two axis gyroscope, a three axis gyroscopeor other multiple axis gyroscope, a one-two-three or multiple axisaccelerometer, a potentiometer, and a MEMS instrument configured toprovide one or more of roll, pitch, yaw, orientation, or vibrationinformation related to the on tool tracking device.
 219. The method ofclaim 217, assessing further comprising: a comparison of data receivedfrom the on tool tracking device and data provided using a computerassisted surgery surgical plan.
 220. The method of claim 217, furthercomprising: determining a predefined computer aided surgery processingmode based on the results of the assessing step, the predefinedprocessing mode selected from the group consisting of a hover mode, siteapproach mode, and active step mode.
 221. The method of claim 217,wherein the step of providing a CAS output to the user is changed and anOTT CAS processing technique or output is modified as a result of theuser performing one or more steps of a computer assisted surgeryprocedure on a knee comprising: making a distal femur cut, making adistal femur anterior cut, making a distal femur posterior lateralcondyle cut, making a distal femur posterior medial condyle cut, makinga distal femur anterior chamfer cut, making a distal femur posteriorlateral condyle chamfer cut, making a distal femur posterior medialcondyle chamfer cut, making the distal femur box cuts, drilling thecavity of a distal femur stabilization post, making a proximal tibialcut, making proximal tibia keel cut, or drilling proximal tibia keel'sholes.
 222. The method of claim 217, wherein the position of the tool iscalculated based on a spatial relationship between the position of thefirst pair of cameras and one or more reference frames attached to apatient and data from the one or more sensors.
 223. The method of claim217, wherein the collecting and processing CAS data using the on tooltracking device further includes receiving data from a second pair ofcameras on or within a housing of the on tool tracking device.
 224. Themethod of claim 223, wherein the position of the tool is calculatedbased on a spatial relationship between the position of the first andsecond pairs of cameras and one or more reference frames attached to apatient and data from the one or more sensors.